Alert: high error rate in vulture#5206
Merged
ruslan-mikhailov merged 3 commits intografana:mainfrom Jun 4, 2025
Merged
Conversation
ruslan-mikhailov
force-pushed
the
vulture-alert
branch
2 times, most recently
from
2328ecb to
aac92e8
Compare
ruslan-mikhailov
requested review from
carles-grafana,
electron0zero,
ie-pham,
javiermolinar,
joe-elliott,
mapno,
mdisibio,
stoewer,
yvrhdn and
zalegrala
as code owners
ruslan-mikhailov
force-pushed
the
vulture-alert
branch
from
aac92e8 to
3ebb7ad
Compare
Contributor
Author
|
+ |
javiermolinar
approved these changes
Jun 4, 2025
|
|
||
| This alert fires when Tempo vulture detects a high error rate (above the configured threshold) while validating write or read paths. It indicates there are problems with trace processing or storage. | ||
|
|
||
| If a trace write failed once, it will continue to fail on read operations as well. In that case, restarting vulture will start over with new traces. |
Contributor
There was a problem hiding this comment.
If a trace write failed once, it will continue to fail on read operations as well. In that case, restarting vulture will start over with new traces.
Do we need that? We could just create a ticket to handle this gracefully
Collaborator
There was a problem hiding this comment.
i don't know how often write failures occur but maybe vulture could retry a few times and, if still unsuccessful, it could just exit itself?
Contributor
Author
There was a problem hiding this comment.
Removed this sentence, created issue: #5210
ruslan-mikhailov
force-pushed
the
vulture-alert
branch
from
3ebb7ad to
6b192b7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does: adds alert for high error rate reported by vulture
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]