Skip to content

[Rhythm] Block-builder improvement to skip WAL#4647

Merged
mdisibio merged 21 commits intografana:mainfrom
mdisibio:bb-nowal
Feb 6, 2025
Merged

[Rhythm] Block-builder improvement to skip WAL#4647
mdisibio merged 21 commits intografana:mainfrom
mdisibio:bb-nowal

Conversation

@mdisibio
Copy link
Copy Markdown
Contributor

@mdisibio mdisibio commented Jan 31, 2025

What this PR does:
Update the block-builder to skip the WAL and go directly from in-memory live traces to the backend block. This is more efficient and the block-builders have much higher throughput. Memory allocs are improved, but total working must now be large enough to contain the full cycle of data. Overall the higher throughput is the most important for operational reasons.

               │ before.txt  │              after.txt              │
               │   sec/op    │   sec/op     vs base                │
BlockBuilder-4   808.0m ± 1%   374.7m ± 2%  -53.63% (p=0.000 n=10)

               │  before.txt  │               after.txt               │
               │     B/s      │     B/s       vs base                 │
BlockBuilder-4   36.17Mi ± 1%   78.41Mi ± 1%  +116.80% (p=0.000 n=10)

               │  before.txt   │              after.txt               │
               │     B/op      │     B/op      vs base                │
BlockBuilder-4   1165.5Mi ± 2%   839.9Mi ± 2%  -27.94% (p=0.000 n=10)

               │ before.txt  │             after.txt              │
               │  allocs/op  │  allocs/op   vs base               │
BlockBuilder-4   5.773M ± 0%   5.417M ± 0%  -6.17% (p=0.000 n=10)

Other changes:

  • Integrate max trace size check into liveTraces. Block-builders don't track trace sizes across cycles so don't need traceSizes and reduces map overhead.
  • Validate block-builder config on startup.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copy link
Copy Markdown
Contributor

@mapno mapno left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this changes, but I have mixed feelings about the live traces iterator. It's nondeterministic too much, makes it less predictable.

Comment thread modules/blockbuilder/tenant_store.go
Comment thread modules/blockbuilder/tenant_store.go Outdated
Comment thread modules/blockbuilder/tenant_store.go
@mdisibio mdisibio merged commit 545ab11 into grafana:main Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants