Allow for a idle timeout

[Fabiosilvero]

Hello,

In jenkins GCP plugin, we can set an idle runner timeout. That permits to keep a runner idle for next jobs but deleting it afterwards, that way a runner won't exist forever.

Is it possible to implement it in Garm ? That way, we can have a min idle runners set to 0 but benefit from idle runners instead of deleting / requesting another one when min idle runners is set to 0 in pool :)

It can be really useful especially in the context of Cloud Providers when you pay per minute your VMs :/

[gabriel-samfira]

Hi @Fabiosilvero

I am not completely sure how the workflow you describe works. Perhaps a few more details or a link to some documentation would help.

GARM right now allows for pools where min-idle-runners are set to 0. In those cases, a runner is spun up on demand when a job is recorded in queued state. Once the job is done, the runner is removed and the pool drops to 0 again. That way you incur no costs except for when a job is actually running. Is that what you mean?

[Fabiosilvero]

Last question, won't that feature cause a reconciliation error when using Garm Operator ? Should we ask them ?

[gabriel-samfira]

This is just an idea at the moment. I sadly have no time to tackle it.

Any such feature is always discussed with the nice folks over at Mercedes-Benz. I also try not to break anything for people, and try to keep things as backwards compatible as possible, or at least add new features as opt-ins. If you don't want to use it, you won't even know it's there.

That being said, what I suggested earlier about consuming events and creating an external service that adapts min-idle-runners dynamically, can be used in conjunction with the current operator. But instead of updating GARM through the API, you can patch the custom resource in the operator, which in turn will update GARM.

But again, I have no cycles to tackle this now.

P.S. The adaptive idle runner thing was requested in the past, so I think we should probably add it. Idle runners are great for providers that don't incur extra costs for runners just sitting around (like Incus and LXD). But for public clouds, it incurs cost. An adaptive version would save money.

[gabriel-samfira]

since a runner will be busy -> Garm spins up a new idle runner -> idle runner gets next job ?

yes. Exactly. Say we start from a state where you don't have any runners, because there were no jobs for a while. The flow would be as follows:

• A set of new jobs come in
• GARM reacts and creates runners
• Jobs finish running and signal the runners can be removed
• GARM removes the runners
• Another set of jobs come in
• GARM decides that there is a need to spin up idle runners (if adaptive runners are enabled) because a trend of activity has been established
• GARM calculates a min-idle-runner setting based on trends in the paste say...X minutes/hours, or uses a default min-idle-runner set by the operator as a base for faster reaction times and adapts it later on based on traffic
• GARM spins up a number of idle runners based on this new setting
• GARM continuously adjusts the value based on activity
• Entity goes idle after a while and no new job is triggered (office hours are done)
• GARM starts an idle timeout. Once reached, runners are scaled down to 0 until activity picks up again

This will allow for eager spin-up of runners in times of activity, reducing wait times for clouds that have really slow spin-up times.

[Fabiosilvero]

Perfect, I'll create the issue in that way :)
I want to say again that it is a wonderful project and thank you for being open to requests like that. I understand the constraints around it.
Since I can't dev even if my life depended on it, I'll try to reach out some coworkers who can and see if they can/want to help in this :)

[gabriel-samfira]

This project started out really basic. It is the tiny community it gathered that made it what it is today. People took the time to speak about their use cases and needs and that helped the project grow in a direction that made it useful for more people. To be honest, I didn't expect it to grow as much as it did (even though it's still fairly under the radar - we didn't really talk about it much, which is okay).

Not everyone needs to code to be a part of a community. Feedback, ideas and testing is equally useful as a PR adding or fixing code. Some ideas and feedback can be added, some cannot due to practical constraints, but that does not mean they are not valid or good. So we encourage people to come with feedback, ask for things. They will always be considered even if we can't really add them due to time constraints.

If people contribute code, that is amazing. If not, that's also fine. Whatever the case, whether it's code or just feedback, we always encourage people to open a discussion or issue first. It's best to talk about things before changes are made.