change: disable legacy overrides by default, add flag to opt in by electron0zero · Pull Request #6741 · grafana/tempo

electron0zero · 2026-03-19T11:19:54Z

What this PR does:

Disable legacy (flat, unscoped) overrides by default. Tempo will refuse to start if legacy overrides are detected in either the main config or per-tenant overrides file.

Added enable_legacy_overrides config option (and -config.enable-legacy-overrides CLI flag) to opt back in temporarily.

If legacy overrides are enabled, a deprecation warning is logged on startup and each time per tenant overrides are loaded.

Breaking Change

Tempo will now error on startup if legacy overrides config is detected. Users must either:

Migrate to the new scoped format (recommended):

Before (legacy):

   overrides:
     ingestion_rate_limit_bytes: 20000000
     ingestion_burst_size_bytes: 20000000
     max_bytes_per_trace: 30000000
     max_traces_per_user: 100000

After (new):

 overrides:
   defaults:
     ingestion:
       rate_limit_bytes: 20000000
       burst_size_bytes: 20000000
       max_traces_per_user: 100000
     global:
       max_bytes_per_trace: 30000000

Similar changes can be done in per tenant overrides file, and see https://grafana.com/docs/tempo/latest/configuration/#overrides for more details.

Opt back in temporarily by setting enable_legacy_overrides: true in the overrides config block or -config.enable-legacy-overrides=true on the CLI. This is a temporary escape hatch - legacy overrides will be removed in a future release.

Which issue(s) this PR fixes:
Fixes #

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Legacy (flat, unscoped) overrides are now disabled by default. Tempo will refuse to start if legacy overrides are detected unless the user sets enable_legacy_overrides: true (or -config.enable-legacy-overrides=true). A deprecation warning is logged when legacy overrides are enabled. The tempo-cli migrate-overrides-config command force-enables legacy overrides since its purpose is to convert them to the new format.

Migrate the super_user per-tenant overrides in the jsonnet microservices config from the flat legacy format to the new scoped format with ingestion and global sections. Regenerate compiled configs.

Copilot

Pull request overview

This PR introduces a breaking change to the overrides subsystem by disabling the legacy (flat/unscoped) overrides format by default, and adds an explicit escape-hatch config/CLI flag to temporarily re-enable it while users migrate to the scoped overrides format.

Changes:

Add enable_legacy_overrides (and -config.enable-legacy-overrides) to opt back into legacy overrides temporarily.
Reject legacy overrides on startup unless explicitly opted in, and emit deprecation warnings.
Update Jsonnet examples/compiled manifests and documentation to use the new scoped overrides structure.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
operations/jsonnet/microservices/config.libsonnet	Updates example overrides content to scoped (`ingestion`/`global`) structure.
operations/jsonnet-compiled/microservices/gen/ConfigMap-tempo-overrides.yaml	Regenerates overrides ConfigMap to scoped format.
operations/jsonnet-compiled/microservices-with-extras/gen/ConfigMap-tempo-overrides.yaml	Regenerates overrides ConfigMap to scoped format (extras variant).
operations/jsonnet-compiled/jsonnetfile.lock.json	Updates pinned Jsonnet dependency revisions.
modules/overrides/runtime_config_overrides_test.go	Updates legacy-path tests to explicitly enable legacy overrides.
modules/overrides/overrides_test.go	Adds coverage asserting legacy overrides are disabled by default.
modules/overrides/overrides.go	Enforces legacy disable-by-default and logs deprecation warning.
modules/overrides/config.go	Adds `EnableLegacyOverrides` config field, YAML wiring, and CLI flag.
docs/sources/tempo/configuration/_index.md	Documents the new `enable_legacy_overrides` option.
cmd/tempo/app/config.go	Updates the startup config warning text for legacy overrides usage.
cmd/tempo-cli/cmd-migrate-overrides-config.go	Forces legacy overrides enabled in the migration command path.

Log a deprecation warning on every per-tenant overrides reload when legacy format is detected, matching the behavior in NewOverrides. Also reject legacy per-tenant overrides when EnableLegacyOverrides is false..

electron0zero · 2026-03-19T15:29:29Z

@codex review please

Copilot

Pull request overview

Disables legacy (flat/unscoped) overrides by default and introduces a config/CLI escape hatch to temporarily allow them, while updating generated manifests and docs accordingly.

Changes:

Add enable_legacy_overrides / -config.enable-legacy-overrides to opt into legacy overrides, otherwise error on startup if legacy format is detected.
Add/adjust tests to cover the new default behavior (legacy rejected unless explicitly enabled).
Update example Jsonnet and generated ConfigMaps to the new scoped overrides format; update docs and changelog for the breaking change.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
operations/jsonnet/microservices/config.libsonnet	Update example overrides to new scoped format (`ingestion`/`global`).
operations/jsonnet-compiled/microservices/gen/ConfigMap-tempo-overrides.yaml	Regenerate overrides ConfigMap to scoped format.
operations/jsonnet-compiled/microservices-with-extras/gen/ConfigMap-tempo-overrides.yaml	Regenerate overrides ConfigMap to scoped format.
operations/jsonnet-compiled/jsonnetfile.lock.json	Bump Jsonnet dependency lock versions.
modules/overrides/runtime_config_overrides_test.go	Update tests for new loader signature + explicitly enable legacy in legacy test case.
modules/overrides/runtime_config_overrides.go	Gate legacy per-tenant override loading behind new `enableLegacy` parameter and error if disabled.
modules/overrides/overrides_test.go	Add tests asserting legacy overrides are rejected by default (static + per-tenant).
modules/overrides/overrides.go	Error on startup when legacy overrides are detected unless explicitly enabled; log deprecation warning.
modules/overrides/config.go	Add `EnableLegacyOverrides` config field, YAML handling, and CLI flag registration.
docs/sources/tempo/configuration/manifest.md	Document `enable_legacy_overrides` in the rendered manifest.
docs/sources/tempo/configuration/_index.md	Document `enable_legacy_overrides` config option.
cmd/tempo/app/config.go	Update legacy overrides warning message and add a migration “Explain”.
cmd/tempo-cli/cmd-migrate-overrides-config.go	Force-enable legacy overrides when running the migration command.
CHANGELOG.md	Add breaking-change entry describing the new default + opt-in flag.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 63453badd2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

stoewer

LGTM. However, some of the bot review comments are actually quite useful. Should we apply them?

…ation log, fix warning message - Reset EnableLegacyOverrides before marshaling migrated config output - Throttle per-tenant legacy deprecation warning to once per 10 minutes - Match config warning Explain text with other error messages

electron0zero · 2026-03-24T20:05:22Z

@codex review please

chatgpt-codex-connector · 2026-03-24T20:15:27Z

Codex Review: Didn't find any major issues. Delightful!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

electron0zero added 2 commits March 19, 2026 15:21

chore: migrate jsonnet overrides from legacy to new scoped format

50d4d4d

Migrate the super_user per-tenant overrides in the jsonnet microservices config from the flat legacy format to the new scoped format with ingestion and global sections. Regenerate compiled configs.

Copilot AI review requested due to automatic review settings March 19, 2026 11:19

Add CHANGELOG.md

e95d4ca

Copilot AI reviewed Mar 19, 2026

View reviewed changes

Comment thread modules/overrides/overrides.go

Comment thread modules/overrides/overrides.go Outdated

Comment thread docs/sources/tempo/configuration/_index.md Outdated

electron0zero added 2 commits March 19, 2026 17:45

fix: warn and reject legacy overrides in per-tenant configs

bee836e

Log a deprecation warning on every per-tenant overrides reload when legacy format is detected, matching the behavior in NewOverrides. Also reject legacy per-tenant overrides when EnableLegacyOverrides is false..

review comment fixes

ca84196

Copilot AI review requested due to automatic review settings March 19, 2026 15:23

make generate-manifest

63453ba

electron0zero marked this pull request as ready for review March 19, 2026 15:23

electron0zero requested review from carles-grafana, ie-pham, javiermolinar, knylander-grafana, mapno, mattdurham, mdisibio, oleg-kozlyuk-grafana, ruslan-mikhailov, stoewer, yvrhdn and zalegrala as code owners March 19, 2026 15:23

Copilot AI reviewed Mar 19, 2026

View reviewed changes

Comment thread modules/overrides/overrides_test.go

Comment thread modules/overrides/runtime_config_overrides.go Outdated

Comment thread cmd/tempo/app/config.go Outdated

chatgpt-codex-connector Bot reviewed Mar 19, 2026

View reviewed changes

Comment thread cmd/tempo-cli/cmd-migrate-overrides-config.go

stoewer reviewed Mar 20, 2026

View reviewed changes

electron0zero requested a review from stoewer March 24, 2026 20:05

stoewer approved these changes Mar 25, 2026

View reviewed changes

mapno approved these changes Mar 25, 2026

View reviewed changes

electron0zero merged commit 4fa2045 into grafana:main Mar 25, 2026
27 checks passed

electron0zero deleted the disable_lagacy_overrides branch March 25, 2026 09:44

This was referenced Mar 25, 2026

tempo-cli: rewrite migrate overrides-config and add per-tenant migration command #6793

Merged

[DOC] Migration guide for Tempo 3.x #6560

Closed

This was referenced Mar 27, 2026

[DOC] Migration guide for Tempo 3.x #6814

Open

[DOCs] Add docs for PRs 6525, 6741, 6403, 6618, and 6359 #6818

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

change: disable legacy overrides by default, add flag to opt in#6741

change: disable legacy overrides by default, add flag to opt in#6741
electron0zero merged 7 commits intografana:mainfrom
electron0zero:disable_lagacy_overrides

electron0zero commented Mar 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

electron0zero commented Mar 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

stoewer left a comment

Uh oh!

electron0zero commented Mar 24, 2026

Uh oh!

chatgpt-codex-connector Bot commented Mar 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

electron0zero commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Breaking Change

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

electron0zero commented Mar 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

stoewer left a comment

Choose a reason for hiding this comment

Uh oh!

electron0zero commented Mar 24, 2026

Uh oh!

chatgpt-codex-connector Bot commented Mar 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

electron0zero commented Mar 19, 2026 •

edited

Loading