Skip to content

Fix Ingesters Occassionally Double Flushing#364

Merged
joe-elliott merged 14 commits intografana:masterfrom
joe-elliott:fake-flushes
Nov 24, 2020
Merged

Fix Ingesters Occassionally Double Flushing#364
joe-elliott merged 14 commits intografana:masterfrom
joe-elliott:fake-flushes

Conversation

@joe-elliott
Copy link
Copy Markdown
Collaborator

@joe-elliott joe-elliott commented Nov 23, 2020

What this PR does:
Occassionally ingesters fail to flush due to a double queuing (#217) issue discovered by @calvernaz. This PR looks to fix that by:

  • Creating a shim in front of the priority queues ExclusiveQueues that prevents multiple items from being queued with the same key
  • Adjusting the flushLoop() to either call .Requeue() or .ClearKey() after flush has failed or succeeded
  • Consolidated helper methods to extract prometheus metric values to ./pkg/util/test
  • Actually wait for flush queues to clear in stopping() eep!

Which issue(s) this PR fixes:
Fixes #217

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott joe-elliott changed the title Fix Bad Flushes Fix Ingster Sometimes Failing to Flush Nov 23, 2020
@joe-elliott joe-elliott changed the title Fix Ingster Sometimes Failing to Flush Fix Ingester Sometimes Failing to Flush Nov 23, 2020
@joe-elliott joe-elliott changed the title Fix Ingester Sometimes Failing to Flush Fix Ingesters Occassionally Double Flushing Nov 23, 2020
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Comment thread pkg/flushqueues/exclusivequeues.go Outdated
Comment thread pkg/flushqueues/exclusivequeues.go Outdated
Copy link
Copy Markdown
Contributor

@mdisibio mdisibio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. A few minor comments mentioned, but more of just questions than recommendations.

Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott joe-elliott requested a review from mdisibio November 24, 2020 02:43
Copy link
Copy Markdown
Contributor

@calvernaz calvernaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I just have a question regarding the clear operation.

Comment thread pkg/flushqueues/exclusivequeues.go Outdated
Comment thread pkg/flushqueues/exclusivequeues.go
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott joe-elliott merged commit 62af44e into grafana:master Nov 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ingesters sometimes fail to flush

4 participants