Add configurable remote_write headers to metrics-generator

[mapno]

What this PR does:

Adds per-tenant configurable headers for remote_write requests in the metrics-generator.

It also adds envvar expansion to the overrides configuration.

Which issue(s) this PR fixes:
Fixes #2975

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

do we know this is true? if it the prom remote storage fails to apply new configs do we know it continues with the old?

updateProcessors at the level above increments a counter metric on failure which i like. metrics_generator_active_processors_update_failed_total. i would love to just have all of this reloaded and counted under the same metric, but that name is pretty opinionated.

[mapno]

do we know this is true? if it the prom remote storage fails to apply new configs do we know it continues with the old?

Yes. If the update fails, the function returns and the original queue is left untouched https://github.com/prometheus/prometheus/blob/main/storage/remote/write.go#L148-L177.

updateProcessors at the level above increments a counter metric on failure which i like. metrics_generator_active_processors_update_failed_total. i would love to just have all of this reloaded and counted under the same metric, but that name is pretty opinionated.

Good point. Added!

[knylander-grafana]

@mapno Will we need doc updates for this?

[knylander-grafana]

@mapno It looks like we'll need docs for authentication and configuration for this feature. I'll add a note to the issue as well.

[ToonTijtgat2]

Dear, thanks for this feature, could it be that the remote_write_headers parameter is not added in the configuration page? https://grafana.com/docs/tempo/latest/configuration/#standard-overrides

[ToonTijtgat2]

Dear

I'm not sure if I'm doing something wrong, but it does not seem to work for me.
I tried with the legacy config:

      metrics_generator_processors:
      - span-metrics
      - service-graphs
      metrics_generator_remote_write_headers:
        tenant-id: prometheus-apps-app-dev```
        
This shows me in the /status/runtime-config the correct settings.
![image](https://github.com/grafana/tempo/assets/109516706/9739ada2-dda1-4d6e-9147-d2bdb14f3dda)

So I started to try with the new config
I placed it in the conf/tempo.yaml
Placed it on runtime-config/overrides.yaml
placed it under conf/overrides.yaml
It always keeps saying defaults is not found on the legacyoverrides.
The config I try to accomplice looks like this.
```overrides.yaml: |
    defaults:
      global:
        max_bytes_per_trace: 1500000
      metrics_generator:
        processors: [service-graphs, span-metrics]
        remote_write_headers:
          tenant-id: prometheus-apps-app-dev```
          
          
Can someone please help me to set this up correctly, or check what went wrong?

Thanks Toon Tijtgat

[ToonTijtgat2]

This also does not work:
conf/tempo.yaml
overrides:
per_tenant_override_config: /conf/overrides.yaml
max_bytes_per_trace: 1500000
metrics_generator_processors:
- span-metrics
- service-graphs
conf/overrides.yaml
overrides:
"tracing-apps-app-dev":
metrics_generator:
remote_write_headers:
tenant-id: prometheus-apps-app-dev
But now it gives per-tenant overrides config type does not match static overrides config type