Skip to content

Flush support #704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jul 28, 2022
Merged

Flush support #704

merged 11 commits into from
Jul 28, 2022

Conversation

gingerwizard
Copy link
Collaborator

Closes #684

We add a Flush() method to a batch to avoid the need for users to accumulate an entire batch in memory. Flushing effectively writes a block - so we move away from a 1:1 mapping of a block per batch. As well as reducing memory usage this has some performance benefits and ensures users don't need to create a new batch every time.

=== RUN   TestNoFlushWithCompression
Alloc = 5 MiB	TotalAlloc = 39 MiB	Sys = 24 MiB	NumGC = 13
Alloc = 10 MiB	TotalAlloc = 78 MiB	Sys = 32 MiB	NumGC = 19
Alloc = 20 MiB	TotalAlloc = 116 MiB	Sys = 36 MiB	NumGC = 22
Alloc = 24 MiB	TotalAlloc = 158 MiB	Sys = 48 MiB	NumGC = 25
Alloc = 34 MiB	TotalAlloc = 199 MiB	Sys = 56 MiB	NumGC = 27
Alloc = 34 MiB	TotalAlloc = 234 MiB	Sys = 56 MiB	NumGC = 29
Alloc = 28 MiB	TotalAlloc = 271 MiB	Sys = 72 MiB	NumGC = 31
Alloc = 51 MiB	TotalAlloc = 318 MiB	Sys = 86 MiB	NumGC = 32
Alloc = 46 MiB	TotalAlloc = 347 MiB	Sys = 86 MiB	NumGC = 33
--- PASS: TestNoFlushWithCompression (0.69s)
=== RUN   TestFlushWithCompression
Alloc = 10 MiB	TotalAlloc = 48 MiB	Sys = 28 MiB	NumGC = 14
Alloc = 10 MiB	TotalAlloc = 70 MiB	Sys = 28 MiB	NumGC = 17
Alloc = 15 MiB	TotalAlloc = 91 MiB	Sys = 28 MiB	NumGC = 19
Alloc = 12 MiB	TotalAlloc = 113 MiB	Sys = 28 MiB	NumGC = 22
Alloc = 10 MiB	TotalAlloc = 134 MiB	Sys = 28 MiB	NumGC = 25
Alloc = 15 MiB	TotalAlloc = 155 MiB	Sys = 28 MiB	NumGC = 27
Alloc = 12 MiB	TotalAlloc = 177 MiB	Sys = 28 MiB	NumGC = 30
Alloc = 9 MiB	TotalAlloc = 198 MiB	Sys = 28 MiB	NumGC = 33
Alloc = 14 MiB	TotalAlloc = 220 MiB	Sys = 28 MiB	NumGC = 35
--- PASS: TestFlushWithCompression (0.38s)

@gingerwizard gingerwizard requested review from ernado and genzgd July 28, 2022 12:53
@gingerwizard gingerwizard self-assigned this Jul 28, 2022
@gingerwizard gingerwizard marked this pull request as ready for review July 28, 2022 19:53
@gingerwizard
Copy link
Collaborator Author

@genzgd for this PR i also added bool support for Int8 - given until 21.8 CH would respond with Int8 @ernado

I test flush on every type with random data to make sure no duplicates - probably unnecessary but wanted some confidence.

@gingerwizard gingerwizard merged commit 562905b into main Jul 28, 2022
@gingerwizard gingerwizard deleted the flush_support branch August 12, 2022 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow batch to be flushed
2 participants