Randomly killed on large volume

[eliphatfs]

When trying to summarize the usage of a large volume (~300TB, with billions of files), the program is Killed after a random while.

By eyeballing the htop stats RES is only up to 6GB (there is 200GB free) when it is killed. I am not sure what is consumed up so it is killed.

./pdu-x86_64-unknown-linux-gnu /vol/ --max-depth 4 --no-sort --progress
(scanned 62876630, total 85365270585344)

Other runs:
(scanned 60223707, total 81912390385664)Killed
(scanned 28444925, total 65608565600256)Killed

[eliphatfs]

It is killed due to out of memory (kmem consumes 184GB and resident set is 16GB at the point of crashing).

I don't know why. Limiting max-depth to 4 there shouldn't be a lot of memory needed.

[KSXGitHub]

Currently, max-depth is only visual. Fixing this would require smarter strategy, such as not saving the nodes beyond max-depth.

[KSXGitHub]

BTW,

with billions of files

(scanned 62876630, total 85365270585344)

That's not "billions", that's trillions.

[eliphatfs]

I think total is number of bytes in the scanned files? It only scanned 63M files when it crashes

[KSXGitHub]

I think total is number of bytes in the scanned files? It only scanned 63M files when it crashes

Oh. You're right. I haven't touch this codebase for a long time.

[KSXGitHub]

@eliphatfs I don't know if this PR would fix your bug. So I'll need you to test it for me.

First, you'll need to install Rust, clone this repo, git checkout move-depth-from-visualizer-to-tree-builder-277, and build it in release mode:

cargo build --bin=pdu --release

Finally, replace ./pdu-x86_64-unknown-linux-gnu in the command you used with ./target/release/pdu and execute it.

./target/release/pdu /vol/ --max-depth 4 --no-sort --progress