Make each ingester use its own key in the ring

[gouthamve]

Using the same key for all ingesters is a huge bottleneck observed by us and others and makes CAS on Consul fail. See the hoops we had to jump through to make it scale: https://grafana.com/blog/2020/02/11/how-were-abusing-hashicorps-consul-at-grafana-labs/

[pracucci]

Maybe it's a crazy idea, but given there's no strong requirement for ring data persistency (ie. you can run Consul in memory and if it's restarted no disaster occurs) have you ever considered using memcached (key with no TTL) for the ring as well?

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

Still valid

[bboreham]

makes CAS on Consul fail

See #3259

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

Still valid

[danielblando]

@gouthamve I just found this issue opened. I have been working with a similar idea after noticing that consul and etcd do have performance degradation when scaling.
It would be nice if you want to give your opinion on it
Proposal:
#4832

Implementation:
#5026