Skip to content

[BugFix] Metrics streaming releases intermediate/incomplete results#5937

Merged
joe-elliott merged 10 commits intografana:mainfrom
joe-elliott:fix-metrics-streaming
Nov 13, 2025
Merged

[BugFix] Metrics streaming releases intermediate/incomplete results#5937
joe-elliott merged 10 commits intografana:mainfrom
joe-elliott:fix-metrics-streaming

Conversation

@joe-elliott
Copy link
Copy Markdown
Collaborator

@joe-elliott joe-elliott commented Nov 12, 2025

What this PR does:
Fixes a situation where intermediate metrics results are released during TraceQL streaming. For most metrics streams this doesn't matter. However, some metrics functions (like topk) when released early will create inconsistent results.

Before:
image

After:
image

The approach taken in this PR was to use the same "shard tracking" functionality currently used for the most_recent query hint. The query_range combiner logic has been updated to only release metrics when we know that all jobs that could impact a particular time range are complete.

Pros:

  • future proof. the results are complete when returned
  • already streamed results don't change value after being returned

Cons:

  • more complex
  • "slower" experience for the user

Streaming Before

before.mp4

Streaming After

after.mp4

Other Changes

  • Updated docker-compose examples to have a streaming and non-streaming datasource

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott joe-elliott force-pushed the fix-metrics-streaming branch from 5ca795d to 1da129c Compare November 12, 2025 18:43
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
… signed joe

Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott joe-elliott merged commit bf1b924 into grafana:main Nov 13, 2025
37 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants