garm http(s) endpoints

[lesaux]

My scenario is the following, I would like to run garm in a stateful set in GKE, and then use the gce_provider for a pool of runners on GCE vms.

When using a github application to authenticate, Are garm public endpoints reachable by github.com still needed? (i.e. The base webhook and the controller webhook URLs) - It seems no endpoints are required in actions-runner-controller for example.

If they are, Is it possible to have different urls for Metadata URL and Callback URL? it feels like these endpoint could be private internal endpoints.

In general, how do you secure these garm public endpoint? Do you use github whitelist ips?

[gabriel-samfira]

Hi @lesaux !

I will try to answer each concern in turn. Apologies for the wall of text.

When using a github application to authenticate, Are garm public endpoints reachable by github.com still needed? (i.e. The base webhook and the controller webhook URLs)

The only endpoint that (currently) needs to be reachable by github is the /webhooks URL. You can view your current webhook URL using:

garm-cli controller show

This endpoint is used by GitHub to send workflow webhooks to GARM, so we know when a new job has started, and that we need to schedule a new ephemeral runner.

If they are, Is it possible to have different urls for Metadata URL and Callback URL? it feels like these endpoint could be private internal endpoints.

The metadata and callback URLs can be changed independently from the webhook URL. You can do so using:

garm-cli controller update --callback-url https://callback.example.com --metadata-url https://metadata.example.com 

There is an explanation of these URLs in the using garm document.

In general, how do you secure these garm public endpoint? Do you use github whitelist ips?

When you add a new repo, org or entrprise, you have the option to install a webhook (or add it manually). The action of adding a webhook also requires a secret. That secret is used to validate all requests that come into the /webhooks endpoint. If the signature of the request cannot be validated against the secret, it is rejected.

The callback and metadata URLs are protected by short lived JWT tokens generated for the instances which will host the github runner. The tokens are valid for 1 hour. Once the instance calls back home with a success or failure status, the token gets invalidated immediately. So, the token is good for at most 1 hour. And it only grants access to that specific runner's information.

---

The only endpoint that needs to be accessible by the outside world, is the /webhooks endpoint. All other API endpoints can be private. In general the requirements are as follows:

• metadata_url - Must be reachable by runners
• callback_url - Must be reachable by runners
• webhook_url - Must be reachable by GitHub

You can chose to limit access to the /webhooks URL exclusively to github (or your instance of GHES) using your firewall. The metadata and callback URLs can be limited to the VMs/containers you spin up. The API endpoints used by garm-cli can be limited to 127.0.0.1 if you wish.

You can find more details here: https://github.com/cloudbase/garm/blob/main/doc/using_garm.md#controller-operations

---

It seems no endpoints are required in actions-runner-controller for example.

GitHubs actions-runner-controllers uses scale sets. This is a feature that uses longpoll to wait for jobs from github. However, the APIs to enable this are undocumented (for now).

I've started working on adding scale sets to GARM, but it is still a work in progress. Future versions of GARM will hopefully have scale sets as well, in which case, the /webhooks endpoint won't be needed unless users prefer to use pools instead of scale sets.

And hopefully, GitHub will document the scale sets APIs (they've been around for more than a year) so we can have somewhat of a guarantee of their stability.

[gabriel-samfira]

Another note of why actions-runner-controller needs no endpoints; GARM is capable of spinning up runners in multiple IaaS at the same time via its providers.

ARC will only spin up runners in the k8s cluster it is installed in. Within the same cluster, runners don't need to phone home. Everything is contained and visible to the controller.

GARM however can spin up runners in EC2, GCP, Openstack, k8s, LXD/Incus, OCI, etc for the same repo, by creating a pool which leverages one of the providers. When those runners come online, they need to fetch their metadata and set their status in GARM.

[ChristopherHX]

And hopefully, GitHub will document the scale sets APIs (they've been around for more than a year) so we can have somewhat of a guarantee of their stability.

Scaleset apis are based on a to be sunset azure pipelines based c# backend, I believe they are going to reimplement them using a new golang framework.

• runner session (work is done to get rid of old apis for months and migration code)
• runner job backend (fully migrated earlier this year)
  • partially in 2024
• started in Q4 2023 with artifacts v4 release

The migration already takes 2 years and counting, it began before scaleset release.

They never documented the legacy runner client api that is a very similar api framework to what I know as scaleset api.

[gabriel-samfira]

Yeah...I kind of gave up on waiting and added scale set support to main. Will migrate once the new APIs are available. Scale sets solve a few problems like more deterministic scheduling, actually being able to schedule runners to a runner group (they don't send the requested runner group name as part of a queued job, like they do with labels), etc.

I had to get scale sets off my mind 😅

[lesaux]

Thank you so much for the very detailed explanation. I will proceed with the setup you describe above, exposing the /webhooks path via a public ingress, and the remainder of the paths on an internal load balancer accessible by my GCE VMs.

With so many public Github actions workflows being shared as containers, arc-DinD mode not performing so well, arc-k8s mode being a bit mysterious, and me not wanting to rewrite all of our current workflows, GARM seems to be the better way!

[gabriel-samfira]

That is precisely why we wrote GARM. ARC is great if a container runtime is sufficient, but a hassle if you need a vm/bm system. The initial beneficiary of GARM was the flatcar linux project. In flatcar we needed to build and test an entire distro. Which meant access to device mapper and other things that are not really namespaced. So we wrote GARM with the idea of allowing runners to spawn wherever we needed them to. And also be simple to use and set up.

Let me know how it works out. We also have a small slack channel if you wish to join. It's in the readme.