Faster restores from cold storage

[philipmw]

Today, rustic supports cold storage through two parameters, --warm-up-command and --warm-up-wait.
rustic calls the warm-up command once per pack, giving it one pack ID at a time.
Then rustic waits for the given duration.
After the wait, rustic assumes that all files are available.

problem

rustic does not know when the packs truly become available for download.

This requires the operator to provide the maximum possible duration that files can take to become available.
If the operator underestimates, rustic will attempt to read an object that isn't available, get an error, and abort the restore.

I want to propose a better way and get consensus before I invest time in implementing it.

The cold storage I am most familiar with is AWS S3 Glacier Deep Archive (GDA).
With GDA, you first request object(s), wait for them to become available, then get them.

There are two key features of S3 that rustic does not currently have support for, but could add it pretty straightforwardly.

The first feature is batch restore.
Instead of requesting one object at a time, S3 supports making a single request with a list of objects.

The second feature is restore notifications.
S3 Deep Archive's restoration can take anywhere from 1 minute to 48 hours.
Instead of waiting a fixed amount, the requester can listen for a notification that the file is ready.

With these two features, restore from Glacier cold storage would be as fast as possible.

It is not possible to achieve this as-is without updating rustic logic.

Batching is not possible because rustic invokes the warm-up command one pack at a time.

Listening for restore notifications is technically possible today, but due to the above, we can wait for only one pack at a time.
For three packs, we'd request one pack, then wait for it to get restored, then request the second pack, wait for that, then continue to the third.

Although today only Glacier supports this, AFAIK, batching and notifications are generic features that any cold storage provider can support. rustic will support these in a generic way.

the key idea

The key shortcoming we'll address is the warm-up command knowing only about one pack at a time, not all packs at once.

We'll create two new restic parameters:

• --warm-up-batch=N (integer argument). The default is 1, with the current behavior of calling the warm-up command with one pack at a time, then waiting for the given time.
  Any larger number N causes restic to invoke the warm-up command with N packs at once.
• --warm-up-pack-id-input=<arg>, with argument being one of anchor or argv. Default is anchor, where rustic replaces the %id anchor in the given warm-up command with the pack ID. Using argv causes rustic to pass the pack ID(s) as argv elements without any string replacement. (We need this because replacing %id cannot meaningfully support batching.)

For Glacier, this set of new parameters will allow the command to batch-request packs and to wait for all those packs to complete restoration.

happy case walkthrough

The user:

• has a cold repo in Glacier Deep Archive;
• configured their cold repo S3 bucket with an SQS queue named rustic-cold-storage-event-queue for S3 event notifications;
• created a dedicated S3 bucket for the restoration manifest, named rustic-restore-manifests;
• own a shell script, restore-from-glacier.sh, that uses the AWS CLI to warm up files. This is a shell script here, but it could be written in many other languages.

The user runs a rustic restore:

rustic restore ... \
  --warm-up-batch=1000 \
  --warm-up-pack-id-input=argv \
  --warm-up-command=./restore-from-glacier.sh \
  --warm-up-wait=0`

Let's say rustic wants to restore 3 packs with IDs pk1, pk2, and pk3. For this, it needs only one batch.
It invokes ./restore-from-glacier.sh pk1 pk2 pk3.
The restore command receives:

argv[0]=./restore-from-glacier.sh
argv[1]=pk1
argv[2]=pk2
argv[3]=pk3

First, the warm-up command creates a batch job manifest and uploads it to S3:

PACKS_TO_RESTORE=$@
MANIFEST_FILE=$(mktemp)
for arg in "$PACKS_TO_RESTORE"; do
  echo "$arg" >> "$MANIFEST_FILE"
done

aws s3api put-object --bucket rustic-restore-manifests --key $MANIFEST_FILE

Next, the warm-up command initiates restoration for all the pack IDs in the manifest. It create a batch job.

aws s3control create-job \
  --region us-west-2 \
  --account-id acct-id \
  --operation '{"S3InitiateRestoreObject": { "ExpirationInDays": 1 }}' \
  --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::rustic-restore-manifests/${MANIFEST_FILE}"}}' \

Once restoration started, the warm-up command starts listening for restoration events:

PACKS_RESTORED=()
QTY_PACKS_TO_RESTORE=...calculate...

while [ $QTY_PACKS_TO_RESTORE -gt 0 ]; do
  MESSAGE_JSON=$(aws sqs receive-message \
    --queue-url https://sqs.us-west-2.amazonaws.com/000111222333/rustic-cold-storage-event-queue \
    --wait-time-seconds 60)

  ...decode message...
  ...add pack to $PACKS_RESTORED...
  QTY_PACKS_TO_RESTORE=...recalculate...
done

Now all files are restored, and restore-from-glacier.sh exits.

rustic resumes execution and starts retrieving cold packs.

progress bar

Today, there is a progress bar during warm-up, updating for each warm-up invocation, so per pack.
With the new system, we would update the progress bar per batch, not per pack.
For most cases where batch size is >= packs we are restoring, the progress bar will be either at 0% or 100%.
This feels like a regression.

The warm-up command for Glacier knows how many packs have finished restoring.
We could devise a protocol for the warm-up command to communicate its status back to rustic.
That would support updating the progress bar more granularly.

I do not want to devise this protocol as a necessary part of this change, as it adds complexity and ambiguity.
The protocol could support more than just the quantity of packs restored, and I want to separate the discussion of potential capabilities and requirements of this protocol.
Once the protocol is devised, we could add it to this feature straightforwardly.

use of argv

Is it ideal to pass all packs in argv, or is there a better way?
I believe it's the best way because it is a POSIX standard, so it is highly portable.

There is a limit on the number of arguments supported in argv. Supposedly the minimum allowed is 4,096. My macOS computer supports 1,048,576.
https://stackoverflow.com/a/7499490/1214939

The limit on argv is main reason I want to support letting the user specify the batch size, rather than just supporting all packs at once.
It is a lower limit than the number of objects Glacier supports in a batch request.

next steps

What do you think about the proposal? Is there a better way? Do you see any concerns?

If I build it as written, will it get accepted?

I would like to build this feature and believe I can, but my main concern is that I am not familiar with Rust.
Would anyone be willing to be my code reviewer?

[philipmw]

@aawsome / @simonsan , tagging you as recent committers. Could you give your thoughts on my proposal please?

[simonsan]

Hey, we are using opendal to access the data, so I would assume everything related tothat storage for a repository needs to be implemented in opendal.

EDIT: The rest i need to check when on my PC the next couple of days, I'm on mobile rn.

[aawsome]

Hi @philipmw!

Sorry, I thought I already answered, but it seems I accidentally did not post my answer - my bad!

So here are some points from memory:

• I think we should discuss details on a PR, I'm happy if you propose one!
• Implementation should be in rustic_core
• refactor: Make new backend when warm_up_command is set. rustic_core#308 might be a good issue to tackle along the way ;-)
• We should not only think about pack files, all file types may need warm-up (see e.g. feat: Add methods for repairing hot/cold repositories rustic_core#255 or feat: Add init --hot-only option #1224 )

@simonsan I already asked to opendal project about support for this (apache/opendal#4308) but so far there is none. The current solution (warm-up-command) does not depend on opendal, so I think a script-based extension might be also valuable for the near future. But we should also think in the direction of direct (i.e. compiled-in) backend support for widely used backends like AWS S3 Glacier. This may also need a refactor/extension of the Backend trait which currently has only a method for warming-up a single file.