Issue running 0.9.2. Listener crash restart loop when running inside ArgoCD

[Scalahansolo]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.9.2

Deployment Method

ArgoCD

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Installed the arc controller and runner set via Helm onto ArgoCD. Once everything came up, the controller would start up a listener that would just get stuck in a crash loop attempting to call out to github and failing GETs.

Describe the bug

When attempting to use the newest 0.9.2 version of the arc controller / runner sets, the listener that was spinning up kept having issues during startup. The screenshot below shows the issue that I was encountering. When downgrading to 0.9.1, the issue went away.

Describe the expected behavior

Additional Context

N/A

Controller Logs

The controller logs had nothing of use in them. They would spin up the listener, see that it died, and tried to spin up a new one. Over and over, forever.

Runner Pod Logs

N/A

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[nikola-jokic]

Hey @Scalahansolo,

Seems like something is canceling the context. Is it possible that something is sending SIGINT to the listener? I tried reproducing the issue, but the listener is up and running.
Can you please try again? Maybe the issue was temporary error on the service side 😕

[carlos-jt]

Same issue was happening too for me yesterday, so i did a rollback to 0.9.0.

[marekaf]

I had the same issue. After rollback one of three listeners doesn't come back up anymore for some spec. merge patch issue.

[Scalahansolo]

@nikola-jokic Im not convinced there was anything wrong with me setup. I can try to bump the version again, but the only change I made to get the listener to stay up was reverting to 0.9.1

[clementnero]

I think the key element here is ArgoCD. AutoscalingListener and associated pod should not be managed by ArgoCD but by the controller directly.
AutoscalingListener resources are never generated by Helm, so ArgoCD will detect these resources as OutOfSync and try to remove them, which explains the SIGINT above.
The problem comes from #3157 in version 0.9.2, which arbitrarily propagates labels to all resources created ... including the tracking label used by ArgoCD, which therefore sees resources created by the controller as having been created by itself, which it tries to delete consecutively as there is no associated definition.

@nikola-jokic a quick solution would be to revert #3157 or add a new setting to disable it.

[Scalahansolo]

That would make sense.

Given that though... the one nice thing about 0.9.2 is that the listener is seen by Argo. For those of us running ArgoCD setups, having K8s assets being spun up that aren't tracked in Argo can be pretty confusing as that's where people go to see the state of the world.

[clementnero]

I agree, but ownerReferences should be added for this purpose. Propagating the ArgoCD tracking label is definitely a blocking issue here.

[xaemiphor]

I picked the wrong day to migrate from the summerwind operator to the gha version, been battling this since yesterday thinking "how did anyone else make this work?" and looking for a toggle in the helm charts.

I can second the issue, #3157 replicating all labels causes argocd to repeatedly try to reap the unexpected AutoscalingListener, Role and RoleBinding created by the operator. Temporary workaround would be disabling auto-prune on the argocd application, but that's not ideal long term.

[manveer-ce]

I also encountered this, on a cluster that only runs ARC.
I had to rollback to 0.9.1 for it to work