Metrics generator crash in 2.10.0

[AndreZiviani]

Describe the bug
After upgrading from 2.9 to 2.10 metrics-generator it started crashing:

panic: Invalid name validation scheme requested: unset
goroutine 797 [running]:
github.com/prometheus/common/model.ValidationScheme.IsValidLabelName(0xc013c06dcf?, {0xc019c6b5c0?, 0x1d?})
        /home/runner/work/tempo/tempo/vendor/github.com/prometheus/common/model/metric.go:203 +0xb4
github.com/prometheus/prometheus/model/relabel.relabel(0xc000039960, 0xc000505900)
        /home/runner/work/tempo/tempo/vendor/github.com/prometheus/prometheus/model/relabel/relabel.go:340 +0x4c5
github.com/prometheus/prometheus/model/relabel.ProcessBuilder(...)
        /home/runner/work/tempo/tempo/vendor/github.com/prometheus/prometheus/model/relabel/relabel.go:290
github.com/prometheus/prometheus/storage/remote.(*QueueManager).StoreSeries(0xc0020be000, {0xc01c800000, 0x874e, 0x41d634?}, 0x0)
        /home/runner/work/tempo/tempo/vendor/github.com/prometheus/prometheus/storage/remote/queue_manager.go:999 +0x30d
github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).readSegment(0xc0020a2780, 0xc00401e000, 0x0, 0x1)
        /home/runner/work/tempo/tempo/vendor/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:520 +0x548
github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).readAndHandleError(0xc0020a2780, 0xc00401e000, 0x0, 0x1, 0x7fffffffffffffff)
        /home/runner/work/tempo/tempo/vendor/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:351 +0x4a
github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).watch(0xc0020a2780, 0x0, 0x1)
        /home/runner/work/tempo/tempo/vendor/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:443 +0x645
github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).Run(0xc0020a2780)
        /home/runner/work/tempo/tempo/vendor/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:319 +0x55a
github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).loop(0xc0020a2780)
        /home/runner/work/tempo/tempo/vendor/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:268 +0x118
created by github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).Start in goroutine 447
        /home/runner/work/tempo/tempo/vendor/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:242 +0xed

To Reproduce
Steps to reproduce the behavior:

1. Install Tempo with tempo-distributed helm chart, I think this is the relevant section:

        metricsGenerator:
          enabled: true
          replicas: 4
          resources:
            requests:
              memory: 3Gi
              cpu: 1000m
          config:
            processor:
              service_graphs:
                max_items: 100000
                dimensions:
                  - deployment.environment
              span_metrics:
                intrinsic_dimensions:
                  status_code: true
                dimensions:
                  - deployment.environment
                  - http.status_code
                  - http.route
                  - rpc.system
                  - rpc.service
                  - rpc.method
                  - rpc.grpc.status_code
              local_blocks:
                filter_server_spans: false
            storage:
              remote_write:
                - url: "http://mimir-gateway.mimir.svc.cluster.local/api/v1/push"
                  write_relabel_configs:
                    - source_labels: [ deployment_environment ]
                      regex: (.*)
                      target_label: client_deployment_environment
        overrides:
          defaults:
            global:
              max_bytes_per_trace: 25000000
            ingestion:
              burst_size_bytes: 50000000
              rate_limit_bytes: 30000000
              max_traces_per_user: 20000
            metrics_generator:
              processors:
                - service-graphs
                - span-metrics
                - local-blocks
              processor:
                span_metrics:
                  enable_target_info: true
                  target_info_excluded_dimensions:
                    - k8s_node_name
                    - k8s_pod_ip
                    - k8s_pod_name
                    - k8s_pod_start_time
                    - k8s_pod_uid
                    - process_command
                    - process_command_args
                    - process_executable_name
                    - process_executable_path
                    - process_owner
                    - process_pid

Expected behavior
Don't crash

Environment:

• Infrastructure: Kubernetes (tempo-distributed)
• Deployment tool: helm

Additional Context
No changes besides upgrading

[AndreZiviani]

full (redacted) helm values:

serviceAccount:
  create: true
  name: tempo
ingester:
  replicas: 9
  maxUnavailable: 1
  resources:
    requests:
      memory: 4Gi
      cpu: 1500m
  persistence:
    enabled: true
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              {{- include "tempo.selectorLabels" (dict "ctx" . "component" "ingester") | nindent 10 }}
          topologyKey: kubernetes.io/hostname
    # force single az
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - us-east-2b

compactor:
  replicas: 4
  maxUnavailable: 1
  config:
    compaction:
      block_retention: 360h # 15d
  resources:
    requests:
      cpu: 1200m
      memory: 4Gi
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              {{- include "tempo.selectorLabels" (dict "ctx" . "component" "compactor") | nindent 10 }}
          topologyKey: kubernetes.io/hostname
    # force single az
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - us-east-2b
distributor:
  replicas: 6
  maxUnavailable: 1
  resources:
    requests:
      memory: 200Mi
      cpu: 600m
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              {{- include "tempo.selectorLabels" (dict "ctx" . "component" "distributor") | nindent 10 }}
          topologyKey: kubernetes.io/hostname
    # force single az
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - us-east-2b
querier:
  replicas: 2
  maxUnavailable: 1
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      cpu: 1000m
      memory: 3Gi
  affinity: |
    # force single az
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - us-east-2b
queryFrontend:
  enabled: true
  replicas: 3
  resources:
    requests:
      cpu: 50m
      memory: 200Mi
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              {{- include "tempo.selectorLabels" (dict "ctx" . "component" "query-frontend") | nindent 10 }}
          topologyKey: kubernetes.io/hostname
    # force single az
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - us-east-2b

multitenancyEnabled: true
streamOverHTTPEnabled: true

traces:
  otlp:
    http:
      enabled: true
    grpc:
      enabled: true

server:
  http_server_read_timeout: 10m
  http_server_write_timeout: 10m
  grpc_server_max_recv_msg_size: 10485760
  grpc_server_max_send_msg_size: 10485760

storage:
  trace:
    block:
      version: vParquet4
    backend: s3
    s3:
      bucket: [REDACTED]
      endpoint: s3.dualstack.us-east-2.amazonaws.com

memcached:
  replicas: 2
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              {{- include "tempo.selectorLabels" (dict "ctx" . "component" "memcached") | nindent 10 }}
          topologyKey: kubernetes.io/hostname
    # force single az
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - us-east-2b

metaMonitoring:
  serviceMonitor:
    enabled: true

gateway:
  enabled: true
  podLabels:
    "vector.dev/exclude": "true"
  replicas: 2
  verboseLogging: false
  ingress:
    enabled: true
    ingressClassName: "traefik"
    annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: private
    hosts:
      - host: [REDACTED]
        paths:
          - path: /
            pathType: Prefix
    tls: {}
  nginxConfig:
    httpSnippet: |
      proxy_read_timeout 300;
      proxy_connect_timeout 300;
      proxy_send_timeout 300;
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              {{- include "tempo.selectorLabels" (dict "ctx" . "component" "gateway") | nindent 10 }}
          topologyKey: kubernetes.io/hostname
    # force single az
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - us-east-2b

metricsGenerator:
  enabled: true
  replicas: 6
  resources:
    requests:
      memory: 3Gi
      cpu: 1000m
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              {{- include "tempo.selectorLabels" (dict "ctx" . "component" "metrics-generator") | nindent 10 }}
          topologyKey: kubernetes.io/hostname
    # force single az
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - us-east-2b
  config:
    processor:
      service_graphs:
        max_items: 100000
        dimensions:
          - deployment.environment
      span_metrics:
        intrinsic_dimensions:
          status_code: true
        dimensions:
          - deployment.environment
          - http.status_code
          - http.route
          - rpc.system
          - rpc.service
          - rpc.method
          - rpc.grpc.status_code
      local_blocks:
        filter_server_spans: false
    storage:
      remote_write:
        - url: "http://mimir-gateway.mimir.svc.cluster.local/api/v1/push"
          write_relabel_configs:
            - source_labels: [ deployment_environment ]
              regex: (.*)
              target_label: client_deployment_environment

overrides:
  defaults:
    global:
      max_bytes_per_trace: 25000000
    ingestion:
      burst_size_bytes: 50000000
      rate_limit_bytes: 30000000
      max_traces_per_user: 20000
    metrics_generator:
      processors:
        - service-graphs
        - span-metrics
        - local-blocks
      processor:
        span_metrics:
          enable_target_info: true
          target_info_excluded_dimensions:
            - k8s_node_name
            - k8s_pod_ip
            - k8s_pod_name
            - k8s_pod_start_time
            - k8s_pod_uid
            - process_command
            - process_command_args
            - process_executable_name
            - process_executable_path
            - process_owner
            - process_pid

[AndreZiviani]

I gave the error to claude and it returned this:

Your concern was valid! I've verified the changes between v2.9.0 and v2.10.0, and my fix is correct. Here's what happened:

What Changed Between v2.9.0 and v2.10.0
1. Prometheus Upgrade:

v2.9.0 used: github.com/prometheus/prometheus v0.304.2
v2.10.0 uses: github.com/prometheus/prometheus v0.307.3

Prometheus added a new NameValidationScheme field to support both UTF-8 and legacy label validation modes. However, this field must be initialized by calling the Validate() method on the RemoteWriteConfig, otherwise it remains as model.UnsetValidation and causes the panic you experienced.

https://github.com/prometheus/prometheus/blob/4321a5573c4400caa7a767aac66fb7a95cf7c0f6/model/relabel/relabel.go#L120-L138

with the patch

diff --git a/modules/generator/storage/config_util.go b/modules/generator/storage/config_util.go
index 51640b9e9..6083b269c 100644
--- a/modules/generator/storage/config_util.go
+++ b/modules/generator/storage/config_util.go
@@ -5,6 +5,7 @@ import (
        "strings"
 
        "github.com/grafana/dskit/user"
+       "github.com/prometheus/common/model"
        prometheus_config "github.com/prometheus/prometheus/config"
 
        "github.com/grafana/tempo/pkg/util"
@@ -52,6 +53,16 @@ func generateTenantRemoteWriteConfigs(inputs []prometheus_config.RemoteWriteConf
                output.SendNativeHistograms = sendNativeHistograms
                output.SendExemplars = true
 
+               // Validate the remote write config to initialize the NameValidationScheme
+               // in all relabel configs. This is required to prevent panics when Prometheus
+               // attempts to use the relabel configs with an unset validation scheme.
+               if err := output.Validate(model.UTF8Validation); err != nil {
+                       logger.Warn("failed to validate remote write config, continuing with potentially unvalidated config",
+                               "remoteWriteName", input.Name,
+                               "remoteWriteURL", input.URL.String(),
+                               "error", err)
+               }
+
                outputs = append(outputs, output)
        }

not sure if it helps or if its correct but looks like it makes sense

[javiermolinar]

I've tried to reproduce this error using the docker compose examples with no luck. Maybe is a different code path but the example already do remote write into prometheus so it should fail every time.

[carles-grafana]

I'm investigating and this is indeed a bug caused by the Prometheus dependency upgrade.

If affects only users using the write_relabel_configs config, I've been able to reproduce it and I'll write a fix.

[carles-grafana]

I've merged a fix: #6399

This will most likely be released with Tempo 2.10.1

[carles-grafana]

fixed in https://github.com/grafana/tempo/releases/tag/v2.10.1