Generators: Drop out of the ring before stopping ingestion

[joe-elliott]

What this PR does:
Currently on shutdown the generators set a flag to stop receiving traffic and then remove themselves from the ring. This causes distributors to continue to send to a generator that is refusing traffic for a few seconds.

This PR:

1. Correctly drops out of the ring first and then sets the flag to stop ingestion
2. Adds a sleep in between these steps. Tempo by default uses memberlist for ring propagation which often requires a few seconds to propagate state from one component to another. Without this sleep the change was ineffective. I do not like the sleep but I'm not sure there's a better solution.

To the left of the red line is what a generator rollout looks like today. To the right is what this change looks like. Note that there are still a few drops but it's significantly reduced.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[mdisibio]

This lgtm. I don't have any qualms about a hard-coded 5s sleep when it helps that much. If it had to be something much longer like 60s to help, then yes agree we would want to look at something else like ring.WaitRingStability.