chunk size relation to s3 cold / hot storage performance

[benjaminpreiss]

Heya everyone!

I am wondering:

1. What is the relation of chunk size and backup / restore performance on hot / cold repos. I reckon that a bigger chunk size will make uploads / downloads faster?
2. Why would I want small chunks?
3. If so, what is the sweet spot regarding the chunk size, for data > 2TB?
4. Are up- and downloads (backup / restore) parallelized?

Anyway, can the chunk size be influenced in rustic?

[aawsome]

Hi @benjaminpreiss

First note that there are 2 different things:

• the chunk size which determines at which point big files are split into chunks - those are then used for the dedupliction. As we use a CDC this is not a fixed size, but rather a size range. Also, for small files we always get "chunks" which are the full file, i.e. the size can be much smaller than the size range. This is general the case for the "last" chunk which is just the remaining of the file even if it is not big enough.
• the pack size which is the size of a pack file - which packs multiple chunks into a file to be saved on the repository storage backend. This again is also just a targeted pack size as the chunks itself are variable in size. And again, packs can be much smaller if there are no more chunks left to pack.

As you are talking about the backend performance, the pack size is IMO the one you are talking about. And yes, that can be influenced, see https://rustic.cli.rs/docs/commands/init/intro.html or use rustic config -h to see the various option.

Chunk sizes can not be changed currently they are set to be in the range of 512kb up to 8MiB.

[benjaminpreiss]

Thank you so much for your detailed answer and the links!

• Would you say that the current default for pack size is the optimum for repos of any size?
• Or should I tweak it for smaller / bigger repos?
• If I expect all the files that I want to backup to be of a specific size, how would that influence my ideal pack size
• What happens if I want to change the pack size, after I have made some backups already (e.g. to cold storage)?

I am just trying to understand what the rules are that govern a good pack size choice 🙏

[aawsome]

@benjaminpreiss Sorry for the late reply:

Would you say that the current default for pack size is the optimum for repos of any size?
Or should I tweak it for smaller / bigger repos?

No, as the "optimum" depends on the storage you are using. But the default tries to implement larger pack files for larger repositories such that the growth in each pack file size and the growth in the number of pack files is roughly the same. If you have a storage where high numbers of files hurt, you should change the default. If you have a storage where (many) very big or very small files hurt, you should change the default.

If I expect all the files that I want to backup to be of a specific size, how would that influence my ideal pack size

No - as said, the pack size is not related to the files you want to backup. Those are anyway chunked and packed together. The pack size determines then only how many of these chunks (depending on the actual sizes of the chunks) are taken and packed together. (Note: there are extreme edge cases - many, many very small files, where this is not 100% true, but these are academical)

What happens if I want to change the pack size, after I have made some backups already (e.g. to cold storage)?

The prune command does repack pack files which are too small or too big. By default, however, only too small pack files unless you change --set-max-packsize-tolerate-percent.