Skip to content

clear metrics generator wal on start up #3548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ie-pham merged 4 commits into from
Apr 19, 2024
Merged

clear metrics generator wal on start up #3548

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4570c7b
clear metrics generator wal on start up
ie-pham Apr 8, 2024
a6c6f9f
add log line
ie-pham Apr 10, 2024
5a75b99
changelogs and move clean wal to even earlier
ie-pham Apr 15, 2024
713340a
clear tenant specific folders only
ie-pham Apr 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
properly. A jsonnet example of an init container is included with the PR.
This impacts impacts all users of the `grafana/tempo` Docker image.
* [CHANGE] Return a less confusing error message to the client when refusing spans due to ingestion rates. [#3485](https://github.com/grafana/tempo/pull/3485) (@ie-pham)
* [CHANGE] Clean Metrics Generator's Prometheus wal before creating instance [#3548](https://github.com/grafana/tempo/pull/3548) (@ie-pham)
* [ENHANCEMENT] Add string interning to TraceQL queries [#3411](https://github.com/grafana/tempo/pull/3411) (@mapno)
* [ENHANCEMENT] Add new (unsafe) query hints for metrics queries [#3396](https://github.com/grafana/tempo/pull/3396) (@mdisibio)
* [ENHANCEMENT] Add nestedSetLeft/Right/Parent instrinsics to TraceQL. [#3497](https://github.com/grafana/tempo/pull/3497) (@joe-elliott)
Expand Down
9 changes: 8 additions & 1 deletion modules/generator/storage/instance.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,19 @@ func New(cfg *Config, o Overrides, tenant string, reg prometheus.Registerer, log

walDir := filepath.Join(cfg.Path, tenant)

// clean the wal before everything
level.Info(logger).Log("msg", "clearing old WAL on start up", "dir", walDir)
Comment thread
joe-elliott marked this conversation as resolved.
err := os.RemoveAll(walDir)
if err != nil {
level.Warn(logger).Log("msg", "failed to remove wal on start up: %s", err.Error())
}

level.Info(logger).Log("msg", "creating WAL", "dir", walDir)

// Create WAL directory with necessary permissions
// This creates both <walDir>/<tenant>/ and <walDir>/<tenant>/wal/. If we don't create the wal
// subdirectory remote storage logs a scary error.
err := os.MkdirAll(filepath.Join(walDir, "wal"), 0o755)
err = os.MkdirAll(filepath.Join(walDir, "wal"), 0o755)
if err != nil {
return nil, fmt.Errorf("could not create directory for metrics WAL: %w", err)
}
Expand Down