Speed up cache size command

[MatthewMckee4]

Summary

uv cache size can be quite slow. Here i use https://github.com/sharkdp/diskus to walk the cache directory with in multiple threads.

Add cli option to set the number of threads and default to std::thread::available_parallelism() or 1.

Test Plan

Added cli statement with info log test.

I believe this is a fair test, where i set cache dir to a large directory.

matthew@matthew-main ~/develop/personal/uv                                                                                                                                                                                                                 [14:17:50]                                                                                                                                                                                                                                       [±cache-size-speed-up ✓▴]
> $ uv cache size --preview-features cache-size -H --cache-dir ~/develop/                                                                                                                                                                   [±cache-size-speed-up ✓▴]
75.7GiB

matthew@matthew-main ~/develop/personal/uv                                                                                                                                                                                                                 [14:18:24]
> $ hyperfine 'uv cache size --preview-features cache-size -H --cache-dir ~/develop/' 'target/debug/uv cache size --preview-features cache-size -H --cache-dir ~/develop/'                                                                  [±cache-size-speed-up ✓▴]
Benchmark 1: uv cache size --preview-features cache-size -H --cache-dir ~/develop/
  Time (mean ± σ):      1.059 s ±  0.014 s    [User: 0.171 s, System: 0.884 s]
  Range (min … max):    1.048 s …  1.097 s    10 runs

Benchmark 2: target/debug/uv cache size --preview-features cache-size -H --cache-dir ~/develop/
  Time (mean ± σ):     413.8 ms ±  17.1 ms    [User: 5789.2 ms, System: 1682.0 ms]
  Range (min … max):   386.3 ms … 441.6 ms    10 runs

Summary
  target/debug/uv cache size --preview-features cache-size -H --cache-dir ~/develop/ ran
    2.56 ± 0.11 times faster than uv cache size --preview-features cache-size -H --cache-dir ~/develop/  

[zanieb]

We can't add Git dependencies, it'll break our crates.io publish

[MatthewMckee4]

Ah, okay, happy for me to copy most of David's code over then?

[zanieb]

I'm not sure yet.

@sharkdp are you interested in publishing the crate? Do you think we should just vendor the parts we need?

[charliermarsh]

I think we should just vendor any speed-ups because this is pulling in a bunch of new dependencies (e.g., a second version of Clap).

[sharkdp]

I updated diskus today (it has always been published on crates.io, so no need for Git dependencies). You should now be able to pull it in as a relatively lightweight dependency using default-features = false. I also made an update specifically for counting apparent file size (which is what uv did here before: sum up metadata.len()), to make that more consistent with what du -sb does (exclude the size of directory entries themselves). I also cleaned up the diskus API a bit. If you want to depend on it here, the code should be something like:

let result = DiskUsage::new(&[cache.root()]).apparent_size().count();
let total_bytes = result.ignore_errors().size_in_bytes();

(or leave out the call to .apparent_size() if you'd rather want to count disk usage).

Note that diskus uses a heuristic for the default number of workers which should ideally not be overwritten for the best possible performance (at least according to benchmarks I did years ago).

I am also completely fine with uv vendoring diskus. It's not a lot of code. In that case, you should potentially make those two updates mentioned above, though.