datanode PVC: No space left on device

[rajarshp]

Hi Team,

we re using Stackable operator 24.7.0 on our Rosa env. But we are seeing one issue, where the data node is giving error that there is n space left on device, though we have enough space left on the PVC.

Could you please let me know if there is any issue with HDFS or it may the an issue with underlying storage system ? If this is an issue with HDFS how we can fix this ?

Error:

2025-03-18 14:56:56,909 WARN datanode.DataNode (BlockReceiver.java:<init>(292)) - IOException in BlockReceiver constructor :Possible disk error: Failed to create /stackable/data/data/datanode/current/XXXXXXX-1741796062629/current/rbw/blk_107XXX52. Cause is
java.io.IOException: No space left on device

[sbernauer]

Hi @rajarshp, can you please provide the output of the df -h command from the datanode?
You might need kubectl exec -it <pod> -- df -h and be quick before the container crashes

[rajarshp]

@sbernauer below is the details.

its look like hdfs generate logs of small sizes which consumes all the inodes rapidly in PVC.
The number of inodes on a volume is 1 inode for every 32 KiB of volume storage capacity.

Please let me know how we can fix this ? for now our PVC has 30GB and it is showing only 3 GB is used.

[sbernauer]

That makes perfect sense, the next guess would have been inodes as well.

I just checked and for a 30Gi pvc on IONOS cloud you get 2.0M, on a different cloud you get 1.9M inodes. So seems like Trident is on the lower side of things with 0.9M.

I googled a bit and found https://kb.netapp.com/Cloud/Astra/Trident/Does_Trident_support_configuring_inodes_on_volume_creation.
I also wanted to mention that you can use multiple PVCs for datanodes, see https://docs.stackable.tech/home/stable/hdfs/usage-guide/resources/#_multiple_storage_volumes.
This could help if you get a higher inode/GiB ratio for smaller persistent volumes.

Anyhow, if you divide 3GB of data by 933887 inodes, you end up with an average file size of 3.37KiB, which is very small.
You can try to fix the current problem as described above, however, I personally would recommend to figure out why there are so many small files and HDFS and try to compact them (e.g. can you use Iceberg's build-in compaction? :) )