ubuntu-24.04-arm: Python Program terminated with signal SIGILL, Illegal instruction

[adrianreber]

Just reported an Illegal instruction problem at https://github.com/orgs/community/discussions/149715. Not sure what is the right location.

[edmorley]

The ubuntu-24.04-arm runner seems to be the cause of this too:
rust-lang/rust#135939

Notably, we've been happily using ARM on GHA for months using the custom runner groups feature, and the errors started immediately after switching to ubuntu-24.04-arm. See:
rust-lang/rust#135939 (comment)

Is there a difference in the ARM image version between the public runner pool and the custom runner pool that could explain this?

[jasonrandrews]

The Ubuntu images are the same, but the hardware is different.
The ubuntu-24.04-arm image uses the Cobalt 100 processor (Neoverse N2), as described in the ChangeLog. The custom runner pool used the previous generation of hardware (Neoverse N1).
Most of the time, illegal instruction is caused by building software for a newer version of the architecture and running it on an older machine, but in this case the failure is on the newer architecture.

[edmorley]

@jasonrandrews Ah thank you for that. I hadn't realised the CPU type was different.

as described in the ChangeLog

The changelog says:

Powered by the Cobalt 100-based processors, these 4 vCPU runners can deliver up to a 40% performance boost compared to Microsoft Azure’s previous generation of Arm-based VMs, providing a power-efficient compute layer for your workloads.

I had seen that, but I think I'd read it as a general "here's what it runs compared to something Azure-specific" not "we've changed/improved the specs since the larger runners feature was launched".

Would it be possible to revise the changelog wording to make that difference clearer? :-)

Also, will the larger runners machines eventually be switched to the Neoverse N2 generation too?

I don't see a mention of the CPU type on either of:

• https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
• https://docs.github.com/en/enterprise-cloud@latest/actions/using-github-hosted-runners/using-larger-runners/about-larger-runners#specifications-for-general-larger-runners

...though I imagine this is intentional since it's typically an implementation detail (we, Heroku, similarly don't list CPU models either).

[jasonrandrews]

Thanks for your patience with the random failures. We discovered it was an issue with the underlying virtual machine and have migrated all the runners to a different arm64 sku (dpdsv5) and this should solve the errors you were seeing. We reported the issues to the Microsoft Azure team, and they are working to debug them, but in the meantime we have migrated our runners so that you can use the runners without issue.

[adrianreber]

I can confirm that it works now. Thanks for fixing it.