Skip to content

bugfix: apply exemplars hint end-to-end and fix safety cap bypass#6559

Merged
zhxiaogg merged 3 commits intografana:mainfrom
zhxiaogg:fix-incorrect-exemplars-hint-handling
Mar 3, 2026
Merged

bugfix: apply exemplars hint end-to-end and fix safety cap bypass#6559
zhxiaogg merged 3 commits intografana:mainfrom
zhxiaogg:fix-incorrect-exemplars-hint-handling

Conversation

@zhxiaogg
Copy link
Copy Markdown
Contributor

@zhxiaogg zhxiaogg commented Feb 25, 2026

What this PR does:

Two bugs in exemplars hint handling (with(exemplars=true|false|N)):

  1. Safety cap bypass: hint value in CompileMetricsQueryRange was not checked against maxExemplars, allowing users to exceed the max limit.

  2. Hint only affected per-shard collection: the frontend combiner always used the config-level MaxExemplars, so the hint never bounded the total exemplars in the final result.

Fix by moving hint resolution into normalizeRequestExemplars at the frontend handler, where req.Exemplars is set once and flows through the sharder, combiner, and backend uniformly.

Which issue(s) this PR fixes:
Fixes CVE-2026-27878

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

if queryRangeReq.Exemplars == 0 || queryRangeReq.Exemplars > cfg.Metrics.Sharder.MaxExemplars {
queryRangeReq.Exemplars = cfg.Metrics.Sharder.MaxExemplars
if err := normalizeRequestExemplars(queryRangeReq, cfg.Metrics.Sharder.MaxExemplars); err != nil {
return httpInvalidRequest(err), nil
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth testing the exception cases, as per annotations

@zhxiaogg zhxiaogg merged commit b13f742 into grafana:main Mar 3, 2026
40 of 41 checks passed
@tempo-ci-app
Copy link
Copy Markdown

tempo-ci-app Bot commented Mar 9, 2026

The backport to release-v2.10 failed:

error cherry-picking: error running git cherry-pick: error running command 'git cherry-pick -x b13f74291d489672601a10297f8fbcbf7dd19192'
error: exit status 1
stdout: Auto-merging CHANGELOG.md
CONFLICT (content): Merge conflict in CHANGELOG.md
Auto-merging modules/frontend/metrics_query_range_handler.go
CONFLICT (content): Merge conflict in modules/frontend/metrics_query_range_handler.go
Auto-merging modules/frontend/metrics_query_range_handler_test.go
CONFLICT (content): Merge conflict in modules/frontend/metrics_query_range_handler_test.go
Auto-merging pkg/traceql/engine_metrics.go
CONFLICT (content): Merge conflict in pkg/traceql/engine_metrics.go
Auto-merging pkg/traceql/engine_metrics_test.go
CONFLICT (content): Merge conflict in pkg/traceql/engine_metrics_test.go

stderr: error: could not apply b13f74291... bugfix: apply exemplars hint end-to-end and fix safety cap bypass (#6559)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

To backport manually, run these commands in your terminal:

git fetch
git switch --create backport-6559-to-release-v2.10 origin/release-v2.10
git cherry-pick -x b13f74291d489672601a10297f8fbcbf7dd19192

Resolve the conflicts, then add the changes and run git cherry-pick --continue:

git add . && git cherry-pick --continue

If you have the GitHub CLI installed:

git push --set-upstream origin backport-6559-to-release-v2.10
PR_BODY=$(gh pr view 6559 --json body --template 'Backport b13f74291d489672601a10297f8fbcbf7dd19192 from #6559{{ "\n\n---\n\n" }}{{ index . "body" }}')
echo "${PR_BODY}" | gh pr create --title '[release-v2.10] bugfix: apply exemplars hint end-to-end and fix safety cap bypass' --body-file - --label 'backport' --label '' --base release-v2.10 --web

Or, if you don't have the GitHub CLI installed (we recommend you install it!):

git push --set-upstream origin backport-6559-to-release-v2.10

And open a pull request where the base branch is release-v2.10 and the compare/head branch is backport-6559-to-release-v2.10

@zhxiaogg zhxiaogg deleted the fix-incorrect-exemplars-hint-handling branch March 9, 2026 15:56
zhxiaogg added a commit to zhxiaogg/tempo that referenced this pull request Mar 10, 2026
zhxiaogg added a commit that referenced this pull request Mar 11, 2026
* enhancement: use frontend MaxExemplars config as single source of truth for exemplar limits (#6515)

(cherry picked from commit 401698d)

* refactor: removed execess exemplars param from CompileMetricsQueryRange (#6538)

(cherry picked from commit 61fafe5)

* bugfix: apply exemplars hint end-to-end and fix safety cap bypass (#6559)

(cherry picked from commit b13f742)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants