Add metrics for bytes and objects written during compaction

[mdisibio]

What this PR does:
Adds three new new metrics to track compaction performance and help estimate capacity. The metrics include a level label which is the input compaction level.

• tempodb_compaction_objects_written and tempodb_compaction_bytes_written which are updated when the compactor flushes to the backend.
• tempodb_compaction_blocks_total Count of input blocks compacted.

Removes tempodb_compaction_duration_seconds metric as it is obsoleted by the new metrics.

Which issue(s) this PR fixes:
n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

Should we drop metricCompactionDuration? I feel like that metric loses it's meaning with the variable block compactions.

Also, what do you think about tempodb_compaction_blocks_total{level=''}. A counter that increments once for every processed block of each level. We could drop the duration histogram while still having some vision on which compactors are compacting blocks at which levels.

Oh, and feel free to add any of this to the operational dashboard that makes sense. I think there's a compaction section now.

[mdisibio]

Should we drop metricCompactionDuration? I feel like that metric loses it's meaning with the variable block compactions.

Also, what do you think about tempodb_compaction_blocks_total{level=''}. A counter that increments once for every processed block of each level. We could drop the duration histogram while still having some vision on which compactors are compacting blocks at which levels.

Oh, and feel free to add any of this to the operational dashboard that makes sense. I think there's a compaction section now.

Agree, added the level label and new metric, and added both new metrics to the operational dashboard. I am not 100% satisfied with sum(increase(tempodb_compaction_blocks_total[5m])) but different sum intervals ($__interval, $__rate_interval, [1m]) lead to poor quality graphs in local testing.

I'm inclined to leave the duration metric for now, while we settle on the best KPIs for compaction.

[joe-elliott]

A couple of final touches and it will be ready to merge. Also make sure to list the removed metric in the changelog. Thanks!