Remove tenant when last block is removed from local backend

[joe-elliott]

What this PR does:
This fixes a long running issue where tenants were persisted even if they had a zero length wal. The issue was that when rediscovering local blocks a tenant was still found even if there were no blocks. This PR fixes the issue with two changes:

• When rediscovering local blocks don't include tenants without blocks
• When clearing a block in the local backend remove the tenant as well

Technically only the first of the two is needed and would be a safer change. Having the second is nice because it will clear up folders in our ingester local backends. Should we remove the second change and err towards safety? Especially with the 2.0 release coming up?

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

Another note:

Currently when rediscovering local blocks this PR will nicely skip tenants that don't have blocks. However, it will leave those folders forever. Only when a block is cleared from the local backend do we make the check on whether or not to remove the tenant. This PR, as is, will not clean up already existing empty tenant folders in the local backend.

[mdisibio]

Only when a block is cleared from the local backend do we make the check on whether or not to remove the tenant. This PR, as is, will not clean up already existing empty tenant folders in the local backend.

Does it make sense to do local folder cleanup at ingester shutdown, which could also handle this scenario?

[joe-elliott]

Does it make sense to do local folder cleanup at ingester shutdown, which could also handle this scenario?

This is likely the better approach. Let me see if I can put something together 👍

[joe-elliott]

@mdisibio i took your suggestion and implemented clean up on shutdown. let me know what you think

[mdisibio]

Forgot this existed. Excellent.

[joe-elliott]

Yeah. It's weird that Shutdown() on the Reader interface does the job of cleaning up, but meh. We should combine these interfaces anyway. I don't think much is gained from the split up.

(or just add Shutdown to the writer?)