Skip to content

Race Condition with handling the updated resource after reconciliation. #1006

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
csviri opened this issue Mar 8, 2022 · 3 comments · Fixed by #1011
Closed

Race Condition with handling the updated resource after reconciliation. #1006

csviri opened this issue Mar 8, 2022 · 3 comments · Fixed by #1011
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@csviri
Copy link
Collaborator

csviri commented Mar 8, 2022

(This part was reworked for 2.2.0 / 3.0.0 already, that is not effected)

@csviri csviri added the kind/bug Categorizes issue or PR as related to a bug. label Mar 8, 2022
@csviri csviri self-assigned this Mar 8, 2022
@andreaTP
Copy link
Collaborator

andreaTP commented Mar 9, 2022

The issue have been detected and reproduced in the Keycloak Operator CI.

Those are the relevant logs: 16_Build (16).txt.
And this is the code for reproducing (under the operator module):
https://github.com/andreaTP/keycloak/tree/check-ci-stability

The sequence of events that is triggering the issue looks as follows:

  • reconciliation in progress
  • receiving an event from k8s (and recociliation is skipped) [event 1]
  • reconciliation finishes and updates the status stanza of the CR
  • receiving an event from k8s because of the modified status (and it gets skipped) [event 2]
  • despite the fact that [event 1] hasn't been processed no reconciliation loop is scheduled
  • following "scheduled reconciliations" (with rescheduleAfter) are ignored too

I do believe that this is a critical bug and we should rise the attention on upgrading in the upcoming release notes.
Thanks @csviri for the support in debugging!

@andreaTP
Copy link
Collaborator

andreaTP commented Mar 9, 2022

For the records:
As a temporary workaround setting generationAwareEventProcessing = false in the ControllerConfiguration prevent the issue from rising.

@andreaTP
Copy link
Collaborator

I confirm that the upcoming SNAPSHOT release fix the issue 🎉
Thanks!

@csviri csviri changed the title Concurrency Issue with handling the updated resource after reconciliation. Race Condition with handling the updated resource after reconciliation. Mar 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants