[Rhythm] Fix ingester to correctly delete old blocks even if not flushing to object storage#5005
Merged
mdisibio merged 4 commits intografana:mainfrom Apr 16, 2025
Conversation
…ject storage. add/update tests
mdisibio
requested review from
carles-grafana,
electron0zero,
ie-pham,
javiermolinar,
joe-elliott,
mapno,
stoewer,
yvrhdn and
zalegrala
as code owners
| } | ||
| break | ||
| // This block can be deleted. | ||
| i.completeBlocks = append(i.completeBlocks[:idx], i.completeBlocks[idx+1:]...) |
Contributor
There was a problem hiding this comment.
Should this line be after the clearBlock try? If we remove the id from the list of complete blocks and then we get an error, the block will be dangling forever
Contributor
Author
There was a problem hiding this comment.
Good catch. I guess that has been an issue for a long time. Also, the previous code was only deleting 1 block per pass, and I attemped to delete all matching blocks. This was broken and needed some other updates because we are modifying the slice while iterating it. Thoughts? There are several ways to do this, and none are particularly clear.
… deletion, fix deleting multiple blocks in 1 pass
mattdurham
reviewed
Apr 15, 2025
joe-elliott
approved these changes
Apr 16, 2025
Collaborator
joe-elliott
left a comment
joe-elliott
left a comment
There was a problem hiding this comment.
a glorious act of software development
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Fixes a bug in #4857 where the ingester doesn't delete blocks off disk when flush_to_storage=false. Additionally, updated the time comparison to look at the block meta to determine the age of the block. Previously it was based on flushed time, which meant that for a retention 1h it wouldn't delete the block off disk until 1h after it was flushed to object storage.
Which issue(s) this PR fixes:
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]