SIGSEGV after update to 0.11.0 on listener

[gordonswing]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.11.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Stay on 0.10.1 on controller and scale set
2. Upgrade to both 0.11.0
3. Check listener logs
4. Revert back to 0.10.1

Describe the bug

During upgrade listener was recreated with the image version 0.11.0 and restarting due to error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1844383]

goroutine 1 [running]:
github.com/actions/actions-runner-controller/cmd/ghalistener/app.New({{0xc0000465e0, 0x1a}, 0x0, 0x0, {0x0, 0x0}, {0xc000430120, 0x28}, {0xc0003053b0, 0x10}, ...})
    github.com/actions/actions-runner-controller/cmd/ghalistener/app/app.go:73 +0x3e3
main.main()
    github.com/actions/actions-runner-controller/cmd/ghalistener/main.go:27 +0x1f8
Stream closed EOF for gha-runners-controller/k8s-dind-6746d74f-listener (listener)

Reverting back to 0.10.1 fixes the issue.
Config wasn't changed during the upgrade.

PS. Controller has no backward compatibility for 0.11.0 version with scale set on 0.10.1. Version should be aligned.

Describe the expected behavior

Smooth upgrade from 0.10.1 to 0.11.0

Additional Context

Ubuntu 24.04.1 LTS (GNU/Linux 6.8.0-45-generic x86_64)
k8s version: v1.31.1+rke2r1

Controller Logs

Not provided

Runner Pod Logs

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1844383]

goroutine 1 [running]:
github.com/actions/actions-runner-controller/cmd/ghalistener/app.New({{0xc0000465e0, 0x1a}, 0x0, 0x0, {0x0, 0x0}, {0xc000430120, 0x28}, {0xc0003053b0, 0x10}, ...})
    github.com/actions/actions-runner-controller/cmd/ghalistener/app/app.go:73 +0x3e3
main.main()
    github.com/actions/actions-runner-controller/cmd/ghalistener/main.go:27 +0x1f8
Stream closed EOF for gha-runners-controller/k8s-dind-6746d74f-listener (listener)

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[jeanmorais]

The same error here.

[AFMiziara]

+1

[jeanmorais]

When I revert to 0.10.1 I got the error:

This self-hosted runner is currently using runner version 2.321.0. This version is out of date. Please update to the latest version 2.323.0

It seems we are locked. Thoughts?

[YuriBucci-Solfacil]

Same error here, i reverted to 0.10.1

[emilio507]

When I revert to 0.10.1 I got the error:

This self-hosted runner is currently using runner version 2.321.0. This version is out of date. Please update to the latest version 2.323.0

It seems we are locked. Thoughts?

This was a similar error I was facing. The fix is to upgrade your runner to version 2.323.0 and from there I was able to get the 0.10.1 to work again.

[emilio507]

One more note, right around the same time that the release of 0.11.0 charts were posted is when the 2.321.0 version of the runner image broke in all of our clusters. It seems some other change was made at the same time which impacted the functionality of the 2.321.0 version of the image making it operable.

[nikola-jokic]

Hey everyone,

The new field for metrics has been added, and I didn't account for it not being present... When you configure metrics, you should specify listenerMetrics, which are configurable set of metrics that will be emitted by the listener.

However, I missed checking if this field is nil since it can be. Thinking about it, it might be a good thing that the listener panics, since you are configuring the listener with metrics, while you didn't specify any. Otherwise, it would silently work, and no metrics would be served.

I think I need to update the release note to call out this field explicitly.

Please let me know if the issue persists when you comment out the listenerMetrics.

For future reference, whenever we have a behavioral change or a breaking change, we issue a minor release. Whenever we have bug fixes, we issue a patch release. This means that when doing minor version upgrades, you should probably read the release notes and make sure you understand what needs to change. I admit I didn't do a good job calling out these major changes, but in the future, please pay attention to the release notes on each minor version upgrade.

[gordonswing]

@nikola-jokic, it would be great to have any breaking changes in ChangeLog.

I have two sets of runners scale-set and none of them have ListenerMetrics enabled:

---
githubConfigUrl: "https://github.com/XXX"
runnerGroup: "dind"
runnerScaleSetName: "k8s-dind"
maxRunners: 30
minRunners: 3

controllerServiceAccount:
  namespace: gha-runners-controller
  name: gha-runner-scale-set-controller

template:
  spec:
    tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "gha-runner"
        effect: "NoSchedule"
    nodeSelector:
      gha-runner: "true"
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: gha-runner
                  operator: In
                  values:
                    - "true"
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        env:
          - name: DOCKER_HOST
            value: unix:///var/run/docker/docker.sock
        securityContext:
          runAsUser: 1001
          runAsGroup: 1001
          privileged: true
          allowPrivilegeEscalation: true
        volumeMounts:
          - name: docker-sock
            mountPath: /var/run/docker
          - name: work
            mountPath: /home/runner/_work
      - name: dind
        image: docker:dind
        args: ["--host=unix:///var/run/docker/docker.sock"]
        securityContext:
          runAsUser: 0
          runAsGroup: 0
          privileged: true
          allowPrivilegeEscalation: true
        lifecycle:
          postStart:
            exec:
              command:
                - sh
                - -c
                - |
                  # Adjust permissions on the socket file
                  while [ ! -S /var/run/docker/docker.sock ]; do
                    sleep 1
                  done
                  chown root:1001 /var/run/docker/docker.sock
                  chmod 660 /var/run/docker/docker.sock
        volumeMounts:
          - name: docker-sock
            mountPath: /var/run/docker
          - name: work
            mountPath: /home/runner/_work
    volumes:
      - name: docker-sock
        emptyDir: {}
      - name: work
        emptyDir: {}
    securityContext:
      fsGroup: 1001

And Second one

---
githubConfigUrl: "https://github.com/XXX"
maxRunners: 30
minRunners: 3
runnerGroup: "k8s"
runnerScaleSetName: "k8s-native"

containerMode:
  type: "kubernetes"

controllerServiceAccount:
  namespace: gha-runners-controller
  name: gha-runner-scale-set-controller

template:
  spec:
    tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "gha-runner"
        effect: "NoSchedule"
    nodeSelector:
      gha-runner: "true"
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: gha-runner
                  operator: In
                  values:
                    - "true"
    securityContext:
      fsGroup: 1001
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        env:
          - name: ACTIONS_RUNNER_CONTAINER_HOOKS
            value: /home/runner/k8s/index.js
          - name: ACTIONS_RUNNER_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
            value: "true"
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
    volumes:
      - name: work
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: ["ReadWriteOnce"]
              storageClassName: "openebs-hostpath"
              resources:
                requests:
                  storage: 10Gi

[nikola-jokic]

Hey @gordonswing,

I completely agree. I should have been more explicit in the release notes, and I will fix that.

The metrics are configured when you install the controller.