Skip to content

Kicbase/ISO: Update buildroot from 2023.02.9 to 2025.2 #20720

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
May 13, 2025
Merged

Kicbase/ISO: Update buildroot from 2023.02.9 to 2025.2 #20720

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f0a5cfa
Kicbase/ISO: Update buildroot from 2023.02.9 to 2024.11.2
ComradeProgrammer May 8, 2025
ff5fb39
Updating ISO to v1.35.0-1746739450-20720
minikube-bot May 9, 2025
5bb96d0
fix restart
ComradeProgrammer May 9, 2025
4f6ad36
fix kvm containerd
ComradeProgrammer May 10, 2025
b7440dc
fix falco-modules
ComradeProgrammer May 10, 2025
4ef9e51
remove falco for good
ComradeProgrammer May 10, 2025
df8794e
use expo retry
ComradeProgrammer May 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion pkg/minikube/cruntime/docker.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,7 @@ func (r *Docker) Enable(disOthers bool, cgroupDriver string, inUserNamespace boo
return err
}

_ = r.Init.ResetFailed("docker")
if err := r.Init.Restart("docker"); err != nil {
return err
}
Expand Down Expand Up @@ -197,11 +198,12 @@ func (r *Docker) Enable(disOthers bool, cgroupDriver string, inUserNamespace boo
return err
}

_ = r.Init.ResetFailed(service)
_ = r.Init.Restart(service)
// try to restart service if stopped, restart until it works
for !r.Init.Active(service) {
fmt.Println("stuck")
time.Sleep(5 * time.Second)
_ = r.Init.ResetFailed(service)
_ = r.Init.Restart(service)
time.Sleep(5 * time.Second)

Expand All @@ -213,6 +215,7 @@ func (r *Docker) Enable(disOthers bool, cgroupDriver string, inUserNamespace boo

// Restart restarts Docker on a host
func (r *Docker) Restart() error {
_ = r.Init.ResetFailed("docker")
return r.Init.Restart("docker")
}

Expand Down
4 changes: 4 additions & 0 deletions pkg/minikube/sysinit/openrc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,10 @@ func (s *OpenRC) Disable(_ string) error {
return nil
}

func (s *OpenRC) ResetFailed(_ string) error {
return nil
}

// DisableNow does Disable + Stop
func (s *OpenRC) DisableNow(svc string) error {
// supposed to do disable + stop
Expand Down
3 changes: 3 additions & 0 deletions pkg/minikube/sysinit/sysinit.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,9 @@ type Manager interface {
// Reload restarts a service
Reload(string) error

// ResetFailed reset the fail counter of a service
ResetFailed(string) error

// Stop stops a service
Stop(string) error

Expand Down
16 changes: 9 additions & 7 deletions pkg/minikube/sysinit/systemd.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,14 +110,16 @@ func (s *Systemd) Restart(svc string) error {
if err := s.daemonReload(); err != nil {
return err
}
// some services declare a realitive small restart-limit in their .service configuration
// so we reset reset-failed counter to override the limit
// and always force restart the service

_, err := s.r.RunCmd(exec.Command("sudo", "systemctl", "restart", svc))
return s.appendJournalctlLogsOnFailure(svc, err)
}

// run systemctl reset-failed for a service
// some services declare a realitive small restart-limit in their .service configuration
// so we reset reset-failed counter to override the limit
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we change the service configuration instead? It will avoid the fake ResetFailed interface we add here.

Copy link
Member Author

@ComradeProgrammer ComradeProgrammer May 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. I guess it is possible, but perhaps we can do this in next PR. Currently buildroot issues are blocking us from building ISO and we cannot update crio, containerd or anything else which involves go>=1.22. It is a rather urgent one

The .service file from cir-dockerd is this cri-docker.service
 where it declares StartLimitBurst=3 StartLimitInterval=60s . I am not sure but I guess this is the problem, because journalctl -u cri-docker.service always shows cri-docker.service: Start request repeated too quickly..

I guess it may also work if we remove these two lines from cri-containerd via go code

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have a burst of start requests?

If the systemd unit is defined properly, system will start the service when dependent service are ready and we should not see such issue.

I guess we install the services dynamically when creating the machine (since we don't know at build time which container runtime will be used). And we probably start them manually without considering the dependencies between services, and then retry failed services?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And we probably start them manually without considering the dependencies between services, and then retry failed services?

I agree. For this issue specifically, accroding to the log here my guessing is that: somehow when we try to start cri-dockerd, the docker daemon/socket is not ready.

However I did tried to wait for docker service/socket with r.Init.Active("docker") before restarting cri-containerd, but it doesn't work at all. r.Init.Active("docker") return true while cri-containerd continue to complain that Cannot connect to the Docker daemon at unix:///var/run/....

So I just came up with this temporary brute-force solution, forcing restart of all those services, and it works. This is definitely not a good idea, I think we should continue to investigate it and see what we can do to actually solve this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However I did tried to wait for docker service/socket with r.Init.Active("docker") before restarting cri-containerd, but it doesn't work at all. r.Init.Active("docker") return true while cri-containerd continue to complain that Cannot connect to the Docker daemon at unix:///var/run/....

is-active is not documented to return true when the service is ready:

   is-active PATTERN...
       Check whether any of the specified units are active (i.e. running).
       Returns an exit code 0 if at least one is active, or non-zero
       otherwise. Unless --quiet is specified, this will also print the
       current unit state to standard output.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this issue specifically, accroding to the log here my guessing is that: somehow when we try to start cri-dockerd, the docker daemon/socket is not ready.

This happens here?

minikube/pkg/minikube/cruntime/docker.go

Line 174 in 8575bee

// restart cri-docker

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, this is exactly the place where it happened.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this behaviour is strange/unexpected i think: we have Type=notify set and NotifyAccess not set, which should mean that the service (ie, its main process) will send the READY=1 signal only when actually "ready", and that should be picked up by the is-active - not sure why it would not work in our setup

ref: https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html

func (s *Systemd) ResetFailed(svc string) error {
_, err := s.r.RunCmd(exec.Command("sudo", "systemctl", "reset-failed", svc))
if err != nil {
return err
}
_, err = s.r.RunCmd(exec.Command("sudo", "systemctl", "restart", svc))
return s.appendJournalctlLogsOnFailure(svc, err)
}

Expand Down