KubeSolo fails on read-only rootfs (Industrial OS / IOT2050): cannot replace /usr/local/bin/runc #74
Description
Introduction
Hey All,
I am working on an IOT development setup and we currently use Docker in combination with Portainer, but we want to make the switch to Kubesolo. At the moment I have a pre-setup Industrial OS setup https://github.com/DGAM-PR/meta-iot2050-DGAM-PR/tree/dgam-pr here, and with this config (added kubesolo via bitbake for adding kubesolo during the compilation of the OS Image) I have kubesolo working from within the OS upon startup. However, I seem to be running into a read-only file system issue where kubesolo tries to write/symlink /usr/local/bin/runc.
Environment
- Device: Siemens IOT2050
- OS: Siemens Industrial OS (Debian-based, read-only root filesystem)
- Rootfs:
/and/usrare read-only;/varand/homeare writable. - KubeSolo version: commit 20dc8b0 (build-date=2025-11-02T22:05:21Z)
Command
kubesolo --portainer-edge-id="yourtown" --portainer-edge-key="locatie"Observed Logs:
2025/11/18 09:24AM INF ../home/runner/work/kubesolo/kubesolo/cmd/kubesolo/main.go:111 > starting kubesolo... | build-date=2025-11-02T22:05:21Z commit=20dc8b0 version=20dc8b0
2025/11/18 09:24AM INF ../home/runner/work/kubesolo/kubesolo/cmd/kubesolo/main.go:113 > ensuring all embedded dependencies are available... | component=kubesolo
2025/11/18 09:24AM FTL ../home/runner/work/kubesolo/kubesolo/cmd/kubesolo/main.go:115 > failed to ensure embedded dependencies | error="failed to load containerd: failed to create symlink for runc: failed to remove existing target CNI config /usr/local/bin/runc: remove /usr/local/bin/runc: read-only file system"
Code-level context
From cmd/kubesolo/main.go:
log.Info().Str("component", "kubesolo").Msg("ensuring all embedded dependencies are available...")
if err := embedded.EnsureEmbeddedDependencies(s.embedded); err != nil {
log.Fatal().Err(err).Msg("failed to ensure embedded dependencies")
}
bootstrap() sets s.embedded to include:
- RuncBinaryFile under the KubeSolo data path (e.g. /var/lib/kubesolo/containerd/runc)
- SystemCNIDir and other system-level paths
Based on the error message, EnsureEmbeddedDependencies attempts to:
- Create a symlink for runc at /usr/local/bin/runc pointing to the embedded runc.
- If something already exists at /usr/local/bin/runc, remove it first.
- On this OS the root filesystem is read-only, so removing /usr/local/bin/runc fails with read-only file system, causing KubeSolo to exit fatally.
runc present but on a different location
On my setup, I do have a runc.. jut not at /usr/local/bin/runc but at /usr/sbin/runc
root@iot2050: which runc
/usr/sbin/runcProblem
On industrial / embedded distributions with read-only root filesystems:
- /usr/local/bin is not writable, by design.
- The system may already ship its own runc (or not), but user-level software usually cannot and should not modify /usr/local/bin.
This makes KubeSolo unusable out of the box on such systems, even though /var/lib/kubesolo and other data paths are writable.
Proposed behavior / ideas (perhaps you have a work arround)
It would be great if EnsureEmbeddedDependencies could:
- Detect a read-only filesystem at /usr/local/bin and:
- Either skip managing /usr/local/bin/runc if a usable runc is already available on $PATH, or
- Fall back to using the embedded runc path directly in the containerd config without requiring the global /usr/local/bin symlink.
- Alternatively, make the target path for the runc symlink configurable (e.g. via a flag/env), so on read-only systems we can keep everything under /var/lib/kubesolo or another writable mount.
- Make the */runc location variable with a flag if that is possible.
Seems Kubesolo has this /usr/local/bin write as a necessity, but I was unable to find a way to make that dependency variable and was not able to find this dependancy either needs to exist or requires write access.
Perhaps someone has a work around I can use.
I currently use bitbake/docker to build the OS and have to pre-install or setup anything that needs to be able to write to a by the IOT device mounted READ-ONLY file system. This also as I will need to reproduce this on potentially 1000's of devices.
Thanks in advance!