Crates that are fetched but not added to the index because of a crash will be lost

[jyn514]

Secondary problem: apparently crates that are fetched but not added to the index will be lost and not build. Relevant code: https://github.com/rust-lang/docs.rs/blob/master/src/docbuilder/queue.rs#L13

Originally posted by @jyn514 in #626 (comment)

crates-index-diff just released a change in 5.1.0 that will let us peek without consuming the changes 🎉 🎉 . This will give us much better durability in the event of a crash, I just need to implement using the new API.

[jyn514]

Mentoring instructions:

• In get_new_crates(), change the line from using fetch_changes() to using peek_changes().
• After adding all new crates to the queue, call index.last_seen_reference().set_target(changes), as per the instructions in the changelog. See also git2::Reference.
• Optionally, change add_crate_to_queue to add multiple crates at once, so the operation will be atomic and we won't add crates that we've already built if we crash halfway through adding crates.

[harrisonbrock]

If this is still open, I would like to work on it.

[jyn514]

@harrisonbrock go for it! If you have questions, feel free to reach out here or on Discord :)

[jyn514]

There was a new API added recently that will make this a little easier: https://docs.rs/crates-index-diff/6.2.0/crates_index_diff/struct.Index.html#method.set_last_seen_reference

Also note that we currently use crates-index-diff 4.0, so it will need to be upgraded to at least 5 (preferably 6) to use this new API.

[harrisonbrock]

Thanks. I have been a little sick and have not had a chance to start yet.

[jyn514]

Get well soon! Your health is more important than code :)

[Byron]

I hope you are doing better, @harrisonbrock!
In case you need additional support related to how to use crates-index-diff, I am happy to help (I have authored crates-index-diff).

[jyn514]

#746 only logged the failure, it will still not be added to the index.