Improve rate limit error message for traces size exceeds rate limit

[electron0zero]

What this PR does:

I saw errors that reads as pusher failed to consume trace data" err="rpc error: code = ResourceExhausted desc = RATE_LIMITED: ingestion rate limit (local: 50000 bytes, global: 500000 bytes) exceeded while adding 966953 bytes for user xxxxxx"

This error message had me confused because the ingestion rate for this user was way under the ingestion rate limit.

After some investigation I noticed that this push failed because the batch size of 966953 bytes is more then the global limit of 500000 bytes (my cluster is using global ingestion rate limiting strategy)

This PR improves the error message when the push failure is due to batch size being more then rate limit and not because user is hitting the ingestion rate limit.

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

does this mean that their batch size is too large? should we advise they reduce it?

[electron0zero]

yes, and yes but decided not to put that in error message because we don't give advice in other error messages as well.

[joe-elliott]

i think it's fine to give more details here. i don't find the new error message any more useful than the old one. you could mention that a single batch exceeded the rate limit.

also, i don't think this check is correct. the push is limited if it exceeds local limit or global limit depending on how tempo is configured. the local limit is just the limit. the global limit is limit / numDistributors. the logic is bit tough to unwind but you can see some if it in ingestion_rate_strategy.go.

[mdisibio]

I'm wondering if what we need is to include the remaining burst size in the error message. I think that is an area that is unclear, because a push can exceed the local limit as a burst. But not when it's consistent.

[electron0zero]

updated to add some more tests and comments to better explain the local and global ingestion rate strategy.

also, i don't think this check is correct. the push is limited if it exceeds local limit or global limit depending on how tempo is configured. the local limit is just the limit. the global limit is limit / numDistributors. the logic is bit tough to unwind but you can see some if it in ingestion_rate_strategy.go.

limit is fetched from ingestionRateLimiter so it will be the local or global limit based on the configured IngestionRateStrategy.

I also updated the check to not include globalLimit because that's not correct, instead we should be checking on limit AND burst to only return the batch size error when the batch is bigger then both limit.

if the batch size is more then limit but less then burst, it would allowed by burst and we should return the ingestion rate limit instead of batch size error.

[zalegrala]

Looks reasonable to me.