Synchronous iterator by mdisibio · Pull Request #2165 · grafana/tempo

mdisibio · 2023-03-03T16:03:11Z

What this PR does:
This PR creates a new synchronous version of the ColumnIterator. Current tests are passing and performance is looking better, but needs a bit more polish and testing. Need to run it against the enhanced set of tests in #2119.

name                                 old time/op    new time/op    delta
BackendBlockTraceQL/noMatch-12         33.8ms ± 2%    31.6ms ± 2%   -6.37%  (p=0.000 n=9+10)
BackendBlockTraceQL/partialMatch-12    53.4ms ±14%    17.0ms ± 1%  -68.19%  (p=0.000 n=10+9)
BackendBlockTraceQL/service.name-12    2.48ms ± 1%    2.43ms ± 3%   -1.94%  (p=0.001 n=10+8)

name                                 old speed      new speed      delta
BackendBlockTraceQL/noMatch-12        505MB/s ± 2%   539MB/s ± 2%   +6.81%  (p=0.000 n=9+10)
BackendBlockTraceQL/partialMatch-12   462MB/s ±13%   667MB/s ± 1%  +44.39%  (p=0.000 n=10+9)
BackendBlockTraceQL/service.name-12   293MB/s ± 1%   299MB/s ± 3%   +1.99%  (p=0.001 n=10+8)

name                                 old MB_io/op   new MB_io/op   delta
BackendBlockTraceQL/noMatch-12           17.0 ± 0%      17.0 ± 0%     ~     (all equal)
BackendBlockTraceQL/partialMatch-12      24.5 ± 0%      11.3 ± 0%  -53.85%  (p=0.000 n=10+10)
BackendBlockTraceQL/service.name-12      0.73 ± 0%      0.73 ± 0%     ~     (all equal)

name                                 old alloc/op   new alloc/op   delta
BackendBlockTraceQL/noMatch-12         2.25MB ± 4%    2.23MB ± 6%     ~     (p=0.400 n=9+10)
BackendBlockTraceQL/partialMatch-12    46.4MB ± 0%    23.4MB ± 1%  -49.52%  (p=0.000 n=10+10)
BackendBlockTraceQL/service.name-12     234kB ± 1%     221kB ± 2%   -5.78%  (p=0.000 n=10+10)

name                                 old allocs/op  new allocs/op  delta
BackendBlockTraceQL/noMatch-12          31.3k ± 0%     31.2k ± 0%   -0.12%  (p=0.000 n=10+8)
BackendBlockTraceQL/partialMatch-12     72.0k ± 0%     49.7k ± 0%  -30.94%  (p=0.000 n=10+10)
BackendBlockTraceQL/service.name-12     3.36k ± 0%     3.32k ± 0%   -1.01%  (p=0.000 n=10+10)

Which issue(s) this PR fixes:
n/a

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Joe Elliott <number101010@gmail.com>

knylander-grafana · 2023-03-28T17:44:45Z

@mdisibio Will we need doc for this?

joe-elliott · 2023-04-14T18:58:58Z

Can we resurrect this PR? I keep seeing traces like this:

Issue here: #2336

This PR would eliminate the possibility that the large gaps seen here are due to synchronization. It would also hopefully allow for easier performance analysis by simplifying this bit of code.

mdisibio · 2023-04-17T14:18:23Z

Can we resurrect this PR?

Yes. A couple updates:

What do you think about putting this behind an environment variable like PARQUET_SYNC_ITERATOR=1? So we can safely merge and test, and also compare performance between pods.
There is still a bug or two that surfaced when using this for the new metrics stuff on real blocks, which isn't caught by the current test suite. Hopefully a simple fix.

joe-elliott · 2023-04-17T18:47:00Z

What do you think about putting this behind an environment variable like PARQUET_SYNC_ITERATOR=1? So we can safely merge and test, and also compare performance between pods.

no concerns. do what makes sense to you. is the PR that risky? or are the gains not very clear?

There is still a bug or two that surfaced when using this for the new metrics stuff on real blocks, which isn't caught by the current test suite. Hopefully a simple fix.

👍

…n SeekTo

…pping out of a partially scanned page, and fix optimization to skip more pages in SeekTo

mdisibio · 2023-04-19T17:52:21Z

@mdisibio Will we need doc for this?

I don't think so. This is a performance experiment and if it pans out we will make it permanent. No user-facing changes.

is the PR that risky? or are the gains not very clear?

A little of both but mostly the former. Concerned about correctness because although our test suite is fairly comprehensive, there are execution paths that are hard to cover synthetically (like jumping between pages, etc). The gains are shaping up better after the last batch of changes, but still not 100% beneficial.

Here are some benchmarks on the wider traceql test suite. Executed by VPARQUET_SYNC_ITERATOR=1 go test ...:

name                                                old time/op    new time/op     delta
BackendBlockTraceQL/spanAttNameNoMatch-12             13.5ms ± 4%     11.3ms ± 4%   -16.29%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttValNoMatch-12              41.9ms ± 3%     38.9ms ± 1%    -7.05%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttValMatch-12                 125ms ±11%       86ms ± 4%   -31.05%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttIntrinsicNoMatch-12        11.2ms ± 7%     10.0ms ±11%      ~     (p=0.056 n=5+5)
BackendBlockTraceQL/spanAttIntrinsicMatch-12           566ms ± 8%      314ms ± 8%   -44.61%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttNameNoMatch-12         2.97ms ±15%     2.52ms ± 6%   -14.97%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttValNoMatch-12          2.84ms ± 2%     2.42ms ± 2%   -14.88%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttValMatch-12            2.86ms ± 2%     2.77ms ±20%      ~     (p=0.548 n=5+5)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12    2.44ms ± 4%     2.38ms ± 4%      ~     (p=0.310 n=5+5)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12       685ms ±33%      227ms ± 7%   -66.83%  (p=0.008 n=5+5)
BackendBlockTraceQL/mixedNameNoMatch-12                2.70s ± 6%      1.72s ± 1%   -36.21%  (p=0.008 n=5+5)
BackendBlockTraceQL/mixedValNoMatch-12                 1.67s ±10%      1.32s ± 1%   -21.01%  (p=0.008 n=5+5)
BackendBlockTraceQL/mixedValMixedMatchAnd-12          2.90ms ±35%     3.32ms ±42%      ~     (p=1.000 n=5+5)
BackendBlockTraceQL/mixedValMixedMatchOr-12            1.97s ±11%      1.53s ± 3%   -22.55%  (p=0.008 n=5+5)
BackendBlockTraceQL/mixedValBothMatch-12              2.38ms ±11%     2.44ms ± 8%      ~     (p=0.421 n=5+5)

name                                                old speed      new speed       delta
BackendBlockTraceQL/spanAttNameNoMatch-12            479MB/s ± 4%    572MB/s ± 3%   +19.43%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttValNoMatch-12             297MB/s ± 3%    320MB/s ± 1%    +7.56%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttValMatch-12              59.9MB/s ±10%   86.7MB/s ± 4%   +44.64%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttIntrinsicNoMatch-12       544MB/s ± 7%    606MB/s ±10%      ~     (p=0.056 n=5+5)
BackendBlockTraceQL/spanAttIntrinsicMatch-12         147MB/s ± 8%    266MB/s ± 8%   +80.77%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttNameNoMatch-12        461MB/s ±14%    540MB/s ± 5%   +17.05%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttValNoMatch-12         479MB/s ± 2%    562MB/s ± 2%   +17.47%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttValMatch-12           476MB/s ± 2%    497MB/s ±18%      ~     (p=0.548 n=5+5)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12   128MB/s ± 4%    131MB/s ± 4%      ~     (p=0.310 n=5+5)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12     115MB/s ±27%    335MB/s ± 6%  +192.53%  (p=0.008 n=5+5)
BackendBlockTraceQL/mixedNameNoMatch-12             9.58MB/s ± 6%  14.99MB/s ± 1%   +56.48%  (p=0.008 n=5+5)
BackendBlockTraceQL/mixedValNoMatch-12              15.5MB/s ± 9%   19.6MB/s ± 1%   +26.18%  (p=0.008 n=5+5)
BackendBlockTraceQL/mixedValMixedMatchAnd-12         481MB/s ±28%    436MB/s ±34%      ~     (p=1.000 n=5+5)
BackendBlockTraceQL/mixedValMixedMatchOr-12         47.0MB/s ±10%   60.5MB/s ± 3%   +28.75%  (p=0.008 n=5+5)
BackendBlockTraceQL/mixedValBothMatch-12             132MB/s ±10%    128MB/s ± 8%      ~     (p=0.421 n=5+5)

name                                                old MB_io/op   new MB_io/op    delta
BackendBlockTraceQL/spanAttNameNoMatch-12               6.44 ± 0%       6.44 ± 0%      ~     (all equal)
BackendBlockTraceQL/spanAttValNoMatch-12                12.4 ± 0%       12.4 ± 0%      ~     (all equal)
BackendBlockTraceQL/spanAttValMatch-12                  7.45 ± 0%       7.45 ± 0%      ~     (all equal)
BackendBlockTraceQL/spanAttIntrinsicNoMatch-12          6.06 ± 0%       6.06 ± 0%      ~     (all equal)
BackendBlockTraceQL/spanAttIntrinsicMatch-12            83.1 ± 0%       83.1 ± 0%      ~     (all equal)
BackendBlockTraceQL/resourceAttNameNoMatch-12           1.36 ± 0%       1.36 ± 0%      ~     (all equal)
BackendBlockTraceQL/resourceAttValNoMatch-12            1.36 ± 0%       1.36 ± 0%      ~     (all equal)
BackendBlockTraceQL/resourceAttValMatch-12              1.36 ± 0%       1.36 ± 0%      ~     (all equal)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12      0.31 ± 0%       0.31 ± 0%      ~     (all equal)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12        76.1 ± 0%       76.1 ± 0%      ~     (all equal)
BackendBlockTraceQL/mixedNameNoMatch-12                 25.8 ± 0%       25.8 ± 0%      ~     (all equal)
BackendBlockTraceQL/mixedValNoMatch-12                  25.8 ± 0%       25.8 ± 0%      ~     (all equal)
BackendBlockTraceQL/mixedValMixedMatchAnd-12            1.36 ± 0%       1.36 ± 0%      ~     (all equal)
BackendBlockTraceQL/mixedValMixedMatchOr-12             92.3 ± 0%       92.3 ± 0%      ~     (all equal)
BackendBlockTraceQL/mixedValBothMatch-12                0.31 ± 0%       0.31 ± 0%      ~     (all equal)

name                                                old alloc/op   new alloc/op    delta
BackendBlockTraceQL/spanAttNameNoMatch-12             1.27MB ±14%     1.27MB ±23%      ~     (p=0.690 n=5+5)
BackendBlockTraceQL/spanAttValNoMatch-12              16.1MB ± 1%     17.6MB ± 5%    +9.84%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttValMatch-12                7.10MB ± 4%     5.86MB ±15%   -17.45%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttIntrinsicNoMatch-12        1.13MB ± 7%     1.36MB ±33%      ~     (p=0.151 n=5+5)
BackendBlockTraceQL/spanAttIntrinsicMatch-12          86.3MB ± 8%     30.3MB ±19%   -64.85%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttNameNoMatch-12          850kB ± 1%     1053kB ±16%   +23.89%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttValNoMatch-12           848kB ± 1%      979kB ± 3%   +15.39%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttValMatch-12             852kB ± 0%     1071kB ± 9%   +25.76%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12     852kB ± 1%      983kB ± 7%   +15.32%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12       117MB ±10%       83MB ± 7%   -29.16%  (p=0.008 n=5+5)
BackendBlockTraceQL/mixedNameNoMatch-12               31.1MB ±37%     29.9MB ±80%      ~     (p=0.381 n=5+5)
BackendBlockTraceQL/mixedValNoMatch-12                40.7MB ±36%     36.4MB ±51%      ~     (p=1.000 n=5+5)
BackendBlockTraceQL/mixedValMixedMatchAnd-12           844kB ± 1%     1039kB ±10%   +23.13%  (p=0.008 n=5+5)
BackendBlockTraceQL/mixedValMixedMatchOr-12            125MB ±49%      57MB ±128%      ~     (p=0.056 n=5+5)
BackendBlockTraceQL/mixedValBothMatch-12               850kB ± 0%      985kB ± 7%   +15.85%  (p=0.008 n=5+5)

name                                                old allocs/op  new allocs/op   delta
BackendBlockTraceQL/spanAttNameNoMatch-12              10.6k ± 0%      10.6k ± 0%    -0.34%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttValNoMatch-12               11.1k ± 0%      10.9k ± 0%    -1.75%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttValMatch-12                 12.0k ± 0%      11.9k ± 0%    -1.13%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttIntrinsicNoMatch-12         10.6k ± 0%      10.6k ± 0%    -0.29%  (p=0.008 n=5+5)
BackendBlockTraceQL/spanAttIntrinsicMatch-12           5.13M ± 0%      0.05M ± 0%   -98.99%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttNameNoMatch-12          10.5k ± 0%      10.5k ± 0%    -0.33%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttValNoMatch-12           10.5k ± 0%      10.5k ± 0%    -0.34%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttValMatch-12             10.5k ± 0%      10.5k ± 0%    -0.33%  (p=0.008 n=5+5)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12     10.6k ± 0%      10.5k ± 0%    -0.28%  (p=0.000 n=5+4)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12       4.31M ± 0%      0.02M ± 0%   -99.54%  (p=0.008 n=5+5)
BackendBlockTraceQL/mixedNameNoMatch-12                12.1k ± 0%      12.6k ± 7%      ~     (p=0.730 n=4+5)
BackendBlockTraceQL/mixedValNoMatch-12                 16.2k ± 2%      16.3k ± 8%      ~     (p=0.548 n=5+5)
BackendBlockTraceQL/mixedValMixedMatchAnd-12           10.5k ± 0%      10.5k ± 0%    -0.33%  (p=0.000 n=5+4)
BackendBlockTraceQL/mixedValMixedMatchOr-12            5.13M ± 0%      0.05M ± 0%   -98.97%  (p=0.029 n=4+4)
BackendBlockTraceQL/mixedValBothMatch-12               10.6k ± 0%      10.5k ± 0%    -0.28%  (p=0.000 n=5+4)

joe-elliott

lgtm. some small comments, but excited to get this one in.

joe-elliott · 2023-04-21T13:00:28Z

+var _ Iterator = (*SyncIterator)(nil)
+
+func NewSyncIterator(ctx context.Context, rgs []pq.RowGroup, column int, columnName string, readSize int, filter Predicate, selectAs string) *SyncIterator {
+


ctx not used? it looks like we're losing the columnIterator.iterate span with this change which i find somewhat useful.

Agree, the span and inspected/kept tags are useful. It's a little atypical but we could start a span in NewSyncIterator and then finish it in iter.Close(). Any other ideas? There's no longer an over-arching method like iterate, and Next/SeekTo are too fine-grained.

oof, good point. our traces need attention generally and this PR is too obvious a win to block.

fine with the "NewSyncIterator" approach to see how it goes.

👍 Tested it out and it works well enough for a first pass or if we have a better idea. The side-effect is that the span starts as soon as the iter is created, instead of on the first "pull" like the async one.

…ted instead of all at the end

mdisibio · 2023-04-24T15:22:34Z

Pushed a small fix to mergeSpanSetIterator to Close() iters as soon as possible when they are exhausted, instead of waiting all the way til the end. Mostly for timing accuracy, performance not really affected.

mdisibio and others added 9 commits March 2, 2023 15:43

Mostly working synchronous iterator and tests

eb40bfc

Fully working version - passing all tempodb/traceql tests

421c4f2

Performance improvements

01feecd

Memmory optimization, don't alloc buffer until needed and use pooling

b681e07

Swap vParquet to the synciterator and a few test tweaks

45aa88c

Merge branch 'main' into sync-iterator

a3c033d

SyncIterator -> Stringer

05bda27

Signed-off-by: Joe Elliott <number101010@gmail.com>

Use dedicated pool/buffers for sync iterator

deced69

Micro-optimization to pass value by address

36f0d67

mdisibio added 9 commits April 18, 2023 09:16

Fix bug where synciterator didn't fully exit after exhausting pages i…

0f91b31

…n SeekTo

Gate parquet synciterator behind an env var

542ad33

Fix bug where synciterator current row number wasn't updated when ski…

0688f56

…pping out of a partially scanned page, and fix optimization to skip more pages in SeekTo

Merge branch 'main' into sync-iterator

9e9db00

Fix bug where synciterator skipped a page when it shouldn't have

d4518f3

lint

3145938

lint

4d440c7

Comments and cleanup

2f00482

Fix OrPredicate.String nil handling

4589863

mdisibio marked this pull request as ready for review April 19, 2023 17:59

mdisibio requested review from annanay25, joe-elliott, mapno, yvrhdn and zalegrala as code owners April 19, 2023 17:59

mdisibio changed the title ~~[WIP] Synchronous iterator~~ Synchronous iterator Apr 19, 2023

Update vparquet2 and changelog

17bd52a

joe-elliott approved these changes Apr 21, 2023

View reviewed changes

mdisibio added 3 commits April 24, 2023 09:16

Apply test updates to vparquet2

477de6a

Fix mergeSpanSetIterator to close sub-iterators as they become exhaus…

ff155e5

…ted instead of all at the end

Add instrumentation to SyncIterator

a610414

mdisibio added 2 commits April 24, 2023 11:23

Merge branch 'main' into sync-iterator

9b2c946

lint

75d336e

mdisibio merged commit 9996433 into grafana:main Apr 25, 2023

mdisibio mentioned this pull request Jun 2, 2023

Make synchronous parquet iterator default #2535

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Synchronous iterator#2165

Synchronous iterator#2165
mdisibio merged 24 commits intografana:mainfrom
mdisibio:sync-iterator

mdisibio commented Mar 3, 2023 •

edited

Loading

Uh oh!

knylander-grafana commented Mar 28, 2023

Uh oh!

joe-elliott commented Apr 14, 2023

Uh oh!

mdisibio commented Apr 17, 2023

Uh oh!

joe-elliott commented Apr 17, 2023

Uh oh!

mdisibio commented Apr 19, 2023 •

edited

Loading

Uh oh!

joe-elliott left a comment

Uh oh!

Uh oh!

joe-elliott Apr 21, 2023

Uh oh!

mdisibio Apr 24, 2023

Uh oh!

joe-elliott Apr 24, 2023

Uh oh!

mdisibio Apr 24, 2023

Uh oh!

Uh oh!

mdisibio commented Apr 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		var _ Iterator = (*SyncIterator)(nil)

		func NewSyncIterator(ctx context.Context, rgs []pq.RowGroup, column int, columnName string, readSize int, filter Predicate, selectAs string) *SyncIterator {

Conversation

mdisibio commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

knylander-grafana commented Mar 28, 2023

Uh oh!

joe-elliott commented Apr 14, 2023

Uh oh!

mdisibio commented Apr 17, 2023

Uh oh!

joe-elliott commented Apr 17, 2023

Uh oh!

mdisibio commented Apr 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joe-elliott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joe-elliott Apr 21, 2023

Choose a reason for hiding this comment

Uh oh!

mdisibio Apr 24, 2023

Choose a reason for hiding this comment

Uh oh!

joe-elliott Apr 24, 2023

Choose a reason for hiding this comment

Uh oh!

mdisibio Apr 24, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mdisibio commented Apr 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mdisibio commented Mar 3, 2023 •

edited

Loading

mdisibio commented Apr 19, 2023 •

edited

Loading