Register all available Go runtime metrics

[andreasgerstmayr]

What this PR does:
Registers all available Go runtime metrics

Which issue(s) this PR fixes:
PR #1528 upgraded prometheus/client_golang to version v1.12.2, which contains the following change:

Reverting addition of new ~80 runtime metrics by default. You can enable this back with GoRuntimeMetricsCollection option or GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection for smooth transition.

The removed metrics contain some interesting GC metrics (e.g. go_gc_cycles_total_gc_cycles_total). This PR (re-)introduces these metrics:

# TYPE go_cgo_go_to_c_calls_calls_total counter
# TYPE go_gc_cycles_automatic_gc_cycles_total counter
# TYPE go_gc_cycles_forced_gc_cycles_total counter
# TYPE go_gc_cycles_total_gc_cycles_total counter
# TYPE go_gc_heap_allocs_by_size_bytes histogram
# TYPE go_gc_heap_allocs_bytes_total counter
# TYPE go_gc_heap_allocs_objects_total counter
# TYPE go_gc_heap_frees_by_size_bytes histogram
# TYPE go_gc_heap_frees_bytes_total counter
# TYPE go_gc_heap_frees_objects_total counter
# TYPE go_gc_heap_goal_bytes gauge
# TYPE go_gc_heap_objects_objects gauge
# TYPE go_gc_heap_tiny_allocs_objects_total counter
# TYPE go_gc_limiter_last_enabled_gc_cycle gauge
# TYPE go_gc_pauses_seconds histogram
# TYPE go_gc_stack_starting_size_bytes gauge
# TYPE go_memory_classes_heap_free_bytes gauge
# TYPE go_memory_classes_heap_objects_bytes gauge
# TYPE go_memory_classes_heap_released_bytes gauge
# TYPE go_memory_classes_heap_stacks_bytes gauge
# TYPE go_memory_classes_heap_unused_bytes gauge
# TYPE go_memory_classes_metadata_mcache_free_bytes gauge
# TYPE go_memory_classes_metadata_mcache_inuse_bytes gauge
# TYPE go_memory_classes_metadata_mspan_free_bytes gauge
# TYPE go_memory_classes_metadata_mspan_inuse_bytes gauge
# TYPE go_memory_classes_metadata_other_bytes gauge
# TYPE go_memory_classes_os_stacks_bytes gauge
# TYPE go_memory_classes_other_bytes gauge
# TYPE go_memory_classes_profiling_buckets_bytes gauge
# TYPE go_memory_classes_total_bytes gauge
# TYPE go_sched_gomaxprocs_threads gauge
# TYPE go_sched_goroutines_goroutines gauge
# TYPE go_sched_latencies_seconds histogram

No currently existing metrics are removed.
This PR is based on grafana/loki#6962.

Alternatively we could also make this configurable if we don't want to export all metrics by default, wdyt?

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[CLAassistant]

All committers have signed the CLA.

[joe-elliott]

Thanks for the addition! I like the idea of adding these metrics, but my impression is that they are rarely used and can be high cardinality.

Can we add a flag to turn these on and have them off by default?

[andreasgerstmayr]

Thanks for the review!

I've moved them to a feature flag now (off by default). I wasn't sure if I should name it enable-go-runtime-metrics, go-runtime-metrics.enabled or more verbose enable-all-go-runtime-metrics. The former sounds more natural for this feature, while the other form is used for components (auth.enabled, search.enabled). Let me know if I should change it, or move the config setting to some other component.

[joe-elliott]

I'm good on the name in the PR. It looks like you have a conflict to resolve?

Heads up that we are going to hold on merging this til after 2.0 is cut. We will likely get an RC out this week and start aggressively working on 2.0. We will revisit merging this likely in a few weeks.

[andreasgerstmayr]

I've rebased the PR and fixed the conflict.

Ok, np - thanks for letting me know. Looking forward to Tempo 2.0 :)

[joe-elliott]

Thanks for the improvement!

[andreasgerstmayr]

Thanks for merging & congrats on the v2 release yesterday! 🎉