Skip to content

[Traceql Metrics] PR 3 - Trace ID sharding#3258

Merged
mdisibio merged 8 commits intomainfrom
traceql-metrics-3-sharding
Jan 12, 2024
Merged

[Traceql Metrics] PR 3 - Trace ID sharding#3258
mdisibio merged 8 commits intomainfrom
traceql-metrics-3-sharding

Conversation

@mdisibio
Copy link
Copy Markdown
Contributor

@mdisibio mdisibio commented Dec 21, 2023

What this PR does:
This PR implements the trace-ID sharding used by metrics in the block Fetch() method. For context: the query-frontend divides a metrics query into jobs along 2 dimensions: horizontally across time (i.e. a job every 5 minutes) and vertically across trace IDs (i.e. 10 shards). This PR is the vertical scaling. For example: shard "10 of 16" processes only traces A00.... through AFF.... This is done with a predicate against the TraceID column.

There are some other details to be aware:

  • Some tricks to shard both 8-byte and 16-byte trace IDs.
  • It uses the previously-experimental trace ID index json in each block. This eliminates quite a few reads because we can go straight to the row groups that contain the matching trace IDs instead of testing every one. I think we fixed the issues with index, it's been working well for me.

Notes
This is one entry in a set of chained PRs. The full body of work has been split into separate buckets to make reviews and updates more manageable.

  1. [Traceql Metrics] PR 1 - Engine #3251
  2. [Traceql Metrics] PR 2 - API #3252
  3. [Traceql Metrics] PR 3 - Trace ID sharding #3258

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@mdisibio mdisibio mentioned this pull request Jan 9, 2024
3 tasks
Base automatically changed from traceql-metrics-2-api to main January 12, 2024 13:48
@mdisibio mdisibio force-pushed the traceql-metrics-3-sharding branch from f79e36b to 08cb7a8 Compare January 12, 2024 14:47
Copy link
Copy Markdown
Contributor

@zalegrala zalegrala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. Nice work.

@mdisibio mdisibio merged commit 3091f69 into main Jan 12, 2024
@mdisibio mdisibio deleted the traceql-metrics-3-sharding branch January 12, 2024 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants