Close WAL block file before deleting everything by kostya9 · Pull Request #2152 · grafana/tempo

kostya9 · 2023-02-27T22:17:42Z

What this PR does: Close WAL block file before deleting everything

Also added some other files closing that I found during a look around the code.

PS: I not proficient in Go, so I could have misunderstood, so please check if it looks good

Which issue(s) this PR fixes:
Attempt to fix #2034
Fixes #2153
Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

CLAassistant · 2023-02-27T22:17:48Z

All committers have signed the CLA.

joe-elliott

thanks for taking a look at this 👍

joe-elliott · 2023-02-28T15:22:32Z

 			}

 			r := parquet.NewReader(file)
+			defer r.Close()


should we also defer close the file above? and all of the files returned from page.file() calls?

parquet.File unfortunately does not implement a Close function, so we can not close it. I noticed that when I was figuring out what is going on and logged an issue - #2153

oh, i forgot this was a parquet.File and not an os.File. wdyt about returning a close() method from file() to close the underlying os file?

Sounds interesting.

What do you think of wrapping the parquet.File with another type that implements Close()? I feel like maybe that would be a more intuitive design. Also, it could later be used in other call sites theoretically (which would probably also need to close the file at some point).

so like:

type pageFile struct { parquet.File os.File }

then the caller can interact with the parquet.file and eventually close the os.file? i'm good on that

Added closing the file opened in file(). It resulted in a little bit more changes that I anticipated, but we should close everything opened in the wal_block module now

Ah, did all that work, but didn't call the final iterator.Close(). Fixed that

joe-elliott

apologies. i had a pending review that i did not submit from a couple days back.

joe-elliott · 2023-03-01T18:56:09Z

 			}

 			r := parquet.NewReader(file)
+			defer r.Close()


oh, i forgot this was a parquet.File and not an os.File. wdyt about returning a close() method from file() to close the underlying os file?

joe-elliott

@mapno I'm good on this, but before merging can you take a quick look? this does a lot of subtle changes with regard to the wal and searching. i'm concerned we may see some strange issues crop up b/c we relied on these files being opened, but i think explicitly closing these files is the right direction.

don't feel the need to do a line by line review (unless you want). mainly flagging you due to the changes along the SearchTags and TagValues paths.

@kostya9 thanks for the persistence and level of detail on this! after @mapno takes a look we will merge.

…ff from the directory Close other files after opening.

…s.File

…MetadataIterator (wrap it instead of spansetIterator)

mapno

LGTM. This change touches a lot of code, so it's difficult to follow all the close() calls. Tests should cover most edge cases. Thanks!

kostya9 · 2023-03-08T11:38:00Z

I see failures in the generator storage tests, from my analysis, the generator doesn't use the code I changed here. Are these tests flaky, or is there a chance something is broken?

joe-elliott · 2023-03-08T13:33:14Z

I see failures in the generator storage tests, from my analysis, the generator doesn't use the code I changed here. Are these tests flaky, or is there a chance something is broken?

Yeah. It's a flakey test. Will rerun.

joe-elliott · 2023-03-08T13:52:34Z

Thanks for the fixes @kostya9!

* WAL_BLOCK should close the file it opened before trying to remove stuff from the directory Close other files after opening. * Add changelog row * Add nullcheck * Add one more missed close * Aggregate errors for wal.Clear * Close file opened by WAL's file() by wrapping the parquet.File with os.File * Add the acual close for the iterator * Fix changelog entry position. Fix types after introduction of spansetMetadataIterator (wrap it instead of spansetIterator)

kostya9 requested review from annanay25, joe-elliott, mapno, mdisibio, yvrhdn and zalegrala as code owners February 27, 2023 22:17

kostya9 mentioned this pull request Feb 27, 2023

Tempo Ingester Flushes Failing #2034

Closed

kostya9 commented Feb 27, 2023

View reviewed changes

Comment thread tempodb/encoding/vparquet/wal_block.go Outdated

kostya9 commented Feb 27, 2023

View reviewed changes

Comment thread tempodb/backend/local/local.go

joe-elliott reviewed Feb 28, 2023

View reviewed changes

kostya9 force-pushed the close_files_after_opening branch from 14268cc to 19a894c Compare March 3, 2023 14:23

joe-elliott reviewed Mar 3, 2023

View reviewed changes

joe-elliott approved these changes Mar 6, 2023

View reviewed changes

Comment thread CHANGELOG.md Outdated

kostya9 added 7 commits March 6, 2023 23:36

WAL_BLOCK should close the file it opened before trying to remove stu…

6fba450

…ff from the directory Close other files after opening.

Add changelog row

8534586

Add nullcheck

ef53365

Add one more missed close

dd2c186

Aggregate errors for wal.Clear

b443093

Close file opened by WAL's file() by wrapping the parquet.File with o…

2eb36be

…s.File

Add the acual close for the iterator

af37d14

kostya9 force-pushed the close_files_after_opening branch from a498555 to 529be34 Compare March 6, 2023 21:39

Fix changelog entry position. Fix types after introduction of spanset…

a3136d7

…MetadataIterator (wrap it instead of spansetIterator)

kostya9 force-pushed the close_files_after_opening branch from 529be34 to a3136d7 Compare March 6, 2023 21:40

mapno approved these changes Mar 8, 2023

View reviewed changes

mapno enabled auto-merge (squash) March 8, 2023 11:24

mapno merged commit 40cc4c5 into grafana:main Mar 8, 2023

kostya9 deleted the close_files_after_opening branch March 8, 2023 15:29

Conversation

kostya9 commented Feb 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Feb 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joe-elliott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

joe-elliott Feb 28, 2023

Choose a reason for hiding this comment

Uh oh!

kostya9 Feb 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joe-elliott Mar 1, 2023

Choose a reason for hiding this comment

Uh oh!

kostya9 Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joe-elliott Mar 3, 2023

Choose a reason for hiding this comment

Uh oh!

kostya9 Mar 3, 2023

Choose a reason for hiding this comment

Uh oh!

kostya9 Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joe-elliott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joe-elliott Mar 1, 2023

Choose a reason for hiding this comment

Uh oh!

joe-elliott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mapno left a comment

Choose a reason for hiding this comment

Uh oh!

kostya9 commented Mar 8, 2023

Uh oh!

joe-elliott commented Mar 8, 2023

Uh oh!

joe-elliott commented Mar 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kostya9 commented Feb 27, 2023 •

edited

Loading

CLAassistant commented Feb 27, 2023 •

edited

Loading

kostya9 Feb 28, 2023 •

edited

Loading

kostya9 Mar 3, 2023 •

edited

Loading

kostya9 Mar 3, 2023 •

edited

Loading