Consolidate backend metrics and add hedged request metric

[josephwoodward]

What this PR does:

This PR does two things:

• Consolidates the backend metrics into a more general purpose metrics
• Adds a new metrics to track the number of hedged requests

Which issue(s) this PR fixes:
Fixes #760

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

Thanks for digging into this. I was thinking more like consolidating the three instrumentation.go files we have in gcs/s3/azure and then rolling this new metric into that.

It's a shame that these metrics names are different but we should keep the metrics backwards compatible:
tempodb_s3_request_duration_seconds
tempodb_azure_request_duration_seconds
tempodb_gcs_request_duration_seconds

All new metrics should not contain the name of the backend and should be the same no matter what backend you choose.

[josephwoodward]

Thanks for digging into this. I was thinking more like consolidating the three instrumentation.go files we have in gcs/s3/azure and then rolling this new metric into that.

It's a shame that these metrics names are different but we should keep the metrics backwards compatible:
tempodb_s3_request_duration_seconds
tempodb_azure_request_duration_seconds
tempodb_gcs_request_duration_seconds

All new metrics should not contain the name of the backend and should be the same no matter what backend you choose.

I see, thanks for the clarification.

So to reiterate: capture duration metrics in a consolidated instrumentation.go (using the current instrumented transport approach) and grab the hedged specific metrics via stats.Snapshot() method every 10 seconds in a Go routine. Is that right? Sorry for all of the questions!

[joe-elliott]

Yup, that's what I was thinking.

[josephwoodward]

Hi @joe-elliott, I've been working on this over the past few evenings and feel it's getting to a point where it'd be valuable to get some feedback on the approach. When you have moment would you mind having a quick look?

Thanks

[joe-elliott]

A well thought out addition. Thank you.

Please add a changelog entry mentioning the new metrics and the deprecation of the old. Other than that this look ready to go.

[josephwoodward]

@joe-elliott This PR is good to go if you're happy with it.

[joe-elliott]

Excellent! Thank you for the contribution.

[josephwoodward]

@joe-elliott No problem! I'm pretty new to Go so this has been a great learning opportunity on a reasonably large, real-world Go product. I'll look at picking up another issue now that this has been merged.