Skip to content

Bloom sharding#192

Merged
annanay25 merged 19 commits intomasterfrom
bloom-sharding
Oct 29, 2020
Merged

Bloom sharding#192
annanay25 merged 19 commits intomasterfrom
bloom-sharding

Conversation

@annanay25
Copy link
Copy Markdown
Contributor

@annanay25 annanay25 commented Sep 30, 2020

Bloom filter sharding is implemented to reduce bloom sizes for larger blocks, this makes them easily cacheable. Slight but not significant reduction in false positive rate is also expected as there is another round of hashing involved.

Fixes #139

Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Copy link
Copy Markdown
Collaborator

@joe-elliott joe-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments. This definitely looks like it's headed in the right direction.

I'm currently unsure on the way to hash into different buckets. Torn between just modding the value and using a real hash function.

Comment thread pkg/bloom/bloom.go
Comment thread pkg/bloom/bloom.go Outdated
Comment thread tempodb/backend/cache/cache.go Outdated
Comment thread tempodb/backend/backend.go Outdated
Comment thread tempodb/backend/util/util.go Outdated
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
@annanay25 annanay25 marked this pull request as ready for review October 16, 2020 15:44
Copy link
Copy Markdown
Collaborator

@joe-elliott joe-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some ci issues. looks like you have some test to fix. one unfortunate comment. apologies for sending you down a bad path.

other than that looks good!

Comment thread pkg/bloom/bloom.go Outdated
Comment thread tempodb/encoding/bloom/bloom.go Outdated
Comment thread tempodb/encoding/bloom/bloom_test.go Outdated
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
@annanay25 annanay25 requested a review from joe-elliott October 28, 2020 16:44
Comment thread tempodb/backend/util/util.go
Copy link
Copy Markdown
Collaborator

@joe-elliott joe-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Remove the check on numTenants
  • Add a changelog entry and note it is a breaking change

patch up these two things let's get this merged :)

Comment thread tempodb/tempodb.go Outdated
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Copy link
Copy Markdown
Collaborator

@joe-elliott joe-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@annanay25 annanay25 merged commit d91c415 into master Oct 29, 2020
@annanay25 annanay25 deleted the bloom-sharding branch October 29, 2020 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce Bloom Filter Sizes

3 participants