Add secure connection support to tempo-cli

[TheoBrigitte]

What this PR does

This PR introduces TLS support for both HTTP and gRPC to tempo-cli query commands, allowing users to connect securely to Tempo instances running with TLS enabled.

Motivation

In production we run Tempo behind a TLS proxy for security. Previously, tempo-cli could only connect using insecure HTTP/gRPC connections, limiting its usefulness in secure environments. This change enables tempo-cli to work with TLS-secured Tempo instances.

Implementation

• Added --secure flag to all tempo-cli query commands:

  • query search
  • query search-tags
  • query search-tag-values
  • query metrics-query-range (both instant and range queries)

• New helper functions in shared.go:

  • httpScheme(secure bool) - Returns "https" or "http" based on secure flag
  • grpcTransportCredentials(secure bool) - Returns appropriate gRPC transport credentials (TLS or insecure)

Implementation Details

• HTTP connections: When --secure is used, HTTP requests use https:// instead of http://
• gRPC connections: When --secure is used, gRPC connections use TLS with system certificate pool instead of insecure credentials
• Backward compatibility: Default behavior remains unchanged (insecure connections) to maintain compatibility with existing usage

Documentation Updates

• Updated docs/sources/tempo/operations/tempo_cli.md to document the new --secure flag for all affected commands

How to test

1. Setup reverse proxy with http2 and tls pointing to service/tempo-query-frontend 3200

Using a Kubernetes Ingress here

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-giantswarm
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  name: tempo
  namespace: tempo
spec:
  ingressClassName: nginx
  rules:
  - host: tempo.host.io
    http:
      paths:
      - backend:
          service:
            name: tempo-query-frontend
            port:
              number: 3200
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - tempo.host.io
    secretName: tempo-ingress-cert

HOST=tempo.host.io
METRICS_QUERY='{ span:name = "GET /:endpoint" } | quantile_over_time(duration, .99) by (span.http.target) with (exemplars=true)'

# Secure HTTP connections

tempo-cli query api metrics           --org-id giantswarm $HOST:443 --secure $METRICS_QUERY "2025-09-29T10:00:00Z" "2025-09-29T11:10:00Z"
tempo-cli query api search-tags       --org-id giantswarm $HOST:443 --secure
tempo-cli query api search-tag-values --org-id giantswarm $HOST:443 --secure .service.name 
tempo-cli query api search            --org-id giantswarm $HOST:443 --secure '{ resource.service.name = "grafana" }' "2025-09-29T10:00:00Z" "2025-09-29T11:10:00Z"

# Secure gRPC connections

tempo-cli query api metrics           --org-id giantswarm $HOST:443 --secure --use-grpc $METRICS_QUERY "2025-09-29T10:00:00Z" "2025-09-29T11:10:00Z"
tempo-cli query api search-tags       --org-id giantswarm $HOST:443 --secure --use-grpc
tempo-cli query api search-tag-values --org-id giantswarm $HOST:443 --secure --use-grpc .service.name 
tempo-cli query api search            --org-id giantswarm $HOST:443 --secure --use-grpc '{ resource.service.name = "grafana" }' "2025-09-29T10:00:00Z" "2025-09-29T11:10:00Z"

Testing

The implementation has been tested with:

• ✅ HTTP connections (both secure and insecure)
• ✅ gRPC connections (both secure and insecure)
• ✅ All query command variants (search, search-tags, search-tag-values, metrics)
• ✅ Backward compatibility (existing behavior unchanged when flag not used)

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[CLAassistant]

All committers have signed the CLA.

[TheoBrigitte]

I am unsure what do to regarding the linting errors for grpc.DialContext, the new grpc.NewClient method uses a different name resolver, but is this a problem here ?

[electron0zero]

I am unsure what do to regarding the linting errors for grpc.DialContext, the new grpc.NewClient method uses a different name resolver, but is this a problem here ?

we should migrate to using grpc.NewClient, I see grpc.NewClient is being used in few places in cmd/tempo-cli/cmd-query-metrics-query-range.go

[TheoBrigitte]

Alright I've replaced those.