Regression: Wrong file owner when using Sysbox CE 0.6.5 or 0.6.6

[smarsching]

After upgrading from Sysbox CE 0.6.3 to 0.6.6, I experienced an issue with file ownership inside the container. I was able to determine that this problem was first introduced with version 0.6.5 (version 0.6.4 still works fine).

The issue can be demonstrated in the following way:

docker run --rm -it --runtime sysbox-runc ubuntu:noble-20250127 ls -l /usr/bin/perl5.38.2
-rwxr-xr-x 2 nobody nogroup 4019312 Apr  5  2024 /usr/bin/perl5.38.2

stat /usr/bin/perl*
  File: /usr/bin/perl
  Size: 3802104   	Blocks: 7432       IO Block: 4096   regular file
Device: 3dh/61d	Inode: 1314501     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-11-23 14:56:46.000000000 +0000
Modify: 2023-11-23 14:56:46.000000000 +0000
Change: 2025-03-28 13:02:32.849864194 +0000
 Birth: 2025-03-28 13:02:32.849864194 +0000
  File: /usr/bin/perl5.34.0
  Size: 3802104   	Blocks: 7432       IO Block: 4096   regular file
Device: 3dh/61d	Inode: 1064129     Links: 2
Access: (0755/-rwxr-xr-x)  Uid: (65534/  nobody)   Gid: (65534/ nogroup)
Access: 2023-11-23 14:56:46.000000000 +0000
Modify: 2023-11-23 14:56:46.000000000 +0000
Change: 2025-03-01 16:21:07.687144619 +0000
 Birth: 2025-03-01 16:21:07.667144958 +0000

When using the regular runc runtime, the problem does not appear:

docker run --rm -it --runtime runc ubuntu:noble-20250127 ls -l /usr/bin/perl5.38.2
-rwxr-xr-x 2 root root 4019312 Apr  5  2024 /usr/bin/perl5.38.2

stat /usr/bin/perl*
  File: /usr/bin/perl
  Size: 3802104   	Blocks: 7432       IO Block: 4096   regular file
Device: 2eh/46d	Inode: 1064129     Links: 2
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-11-23 14:56:46.000000000 +0000
Modify: 2023-11-23 14:56:46.000000000 +0000
Change: 2025-03-01 16:21:07.687144619 +0000
 Birth: 2025-03-01 16:21:07.667144958 +0000
  File: /usr/bin/perl5.34.0
  Size: 3802104   	Blocks: 7432       IO Block: 4096   regular file
Device: 2eh/46d	Inode: 1064129     Links: 2
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-11-23 14:56:46.000000000 +0000
Modify: 2023-11-23 14:56:46.000000000 +0000
Change: 2025-03-01 16:21:07.687144619 +0000
 Birth: 2025-03-01 16:21:07.667144958 +0000

I am not entirely sure why the file /usr/bin/perl5.38.2 (and /usr/bin/gunzip). I suspect that the problem is related to the fact that these are hard links. With the regular runc runtime, both paths correctly refer to the same inode, but with sysbox-runc, the inode ID differs and the ownership differs as well.

When inspecting the image on the host, the ownership seems to be correct:

ls -l /var/lib/docker/overlay2/b859e9bbadcf1093e5fcfd997d4719de4ba633ec97b1517b46fd68a8b3d71c07/diff/usr/bin/perl5.38.2 
-rwxr-xr-x 2 root root 4019312 Apr  5  2024 /var/lib/docker/overlay2/b859e9bbadcf1093e5fcfd997d4719de4ba633ec97b1517b46fd68a8b3d71c07/diff/usr/bin/perl5.38.2

stat /var/lib/docker/overlay2/b859e9bbadcf1093e5fcfd997d4719de4ba633ec97b1517b46fd68a8b3d71c07/diff/usr/bin/perl*
  File: /var/lib/docker/overlay2/b859e9bbadcf1093e5fcfd997d4719de4ba633ec97b1517b46fd68a8b3d71c07/diff/usr/bin/perl
  Size: 4019312   	Blocks: 7856       IO Block: 4096   regular file
Device: fd01h/64769d	Inode: 1069581     Links: 2
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-04-05 21:57:12.000000000 +0200
Modify: 2024-04-05 21:57:12.000000000 +0200
Change: 2025-03-01 17:21:10.127103244 +0100
 Birth: 2025-03-01 17:21:10.099103719 +0100
  File: /var/lib/docker/overlay2/b859e9bbadcf1093e5fcfd997d4719de4ba633ec97b1517b46fd68a8b3d71c07/diff/usr/bin/perl5.38.2
  Size: 4019312   	Blocks: 7856       IO Block: 4096   regular file
Device: fd01h/64769d	Inode: 1069581     Links: 2
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-04-05 21:57:12.000000000 +0200
Modify: 2024-04-05 21:57:12.000000000 +0200
Change: 2025-03-01 17:21:10.127103244 +0100
 Birth: 2025-03-01 17:21:10.099103719 +0100

[href]

We have the same issue, currently pulling debian:bookworm and running apt-get upgrade fails when using Sysbox. That being said, we use 0.6.3, where it fails as well.

[ctalledo]

Hi @smarsching, thanks for filing the issue.

I am not able to reproduce it though on my Ubuntu Jammy host with Linux kernel 6.8, Docker version 28.0.1, and Sysbox 0.6.6:

$ docker run --rm -it --runtime sysbox-runc ubuntu:noble-20250127 bash

root@a44b375eac55:/# ls -l /usr/bin/perl5.38.2
-rwxr-xr-x 2 root root 4019312 Apr  5  2024 /usr/bin/perl5.38.2

root@a44b375eac55:/# stat /usr/bin/perl*

  File: /usr/bin/perl
  Size: 4019312         Blocks: 7856       IO Block: 4096   regular file
Device: 0,104   Inode: 30816542    Links: 2
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-04-05 19:57:12.000000000 +0000
Modify: 2024-04-05 19:57:12.000000000 +0000
Change: 2025-04-15 02:25:59.160467243 +0000
 Birth: 2025-04-15 02:25:59.148467370 +0000

  File: /usr/bin/perl5.38.2
  Size: 4019312         Blocks: 7856       IO Block: 4096   regular file
Device: 0,104   Inode: 30816542    Links: 2
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-04-05 19:57:12.000000000 +0000
Modify: 2024-04-05 19:57:12.000000000 +0000
Change: 2025-04-15 02:25:59.160467243 +0000
 Birth: 2025-04-15 02:25:59.148467370 +0000

What host OS, kernel, and Docker version are you using?

this problem was first introduced with version 0.6.5

Strange, I don't see anything in the 0.6.5 changelog that would cause this (but I believe you).

[ctalledo]

Hi @href,

We have the same issue, currently pulling debian:bookworm and running apt-get upgrade fails when using Sysbox. That being said, we use 0.6.3, where it fails as well.

Likely a different issue, but I can 't repro either:

$ docker run --runtime=sysbox-runc -it --rm debian:bookworm

root@451a07de66e8:/# apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@451a07de66e8:/# apt-get update
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8792 kB]
Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [512 B]
Get:6 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [254 kB]
Fetched 9301 kB in 1s (7819 kB/s)                       
Reading package lists... Done

[href]

Maybe you got a new/different debian:bookworm than I do? Though I cannot get a newer one from the looks of it. Here are the complete set of steps with which I can reproduce this currently:

Be sure that there is an upgradable package (perl, which will try to create the backup link that fails):

~# docker run --rm -it debian@sha256:00cd074b40c4d99ff0c24540bdde0533ca3791edcdac0de36d6b9fb3260d89e2
root@681b04feebbe:/# apt update
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8792 kB]
Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [512 B]
Get:6 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [254 kB]
Fetched 9301 kB in 1s (7960 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.

Run the upgrade:

root@681b04feebbe:/# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  perl-base
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1609 kB of archives.
After this operation, 1024 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://deb.debian.org/debian-security bookworm-security/main amd64 perl-base amd64 5.36.0-7+deb12u2 [1609 kB]
Fetched 1609 kB in 0s (18.7 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 6091 files and directories currently installed.)
Preparing to unpack .../perl-base_5.36.0-7+deb12u2_amd64.deb ...
Unpacking perl-base (5.36.0-7+deb12u2) over (5.36.0-7+deb12u1) ...
dpkg: error processing archive /var/cache/apt/archives/perl-base_5.36.0-7+deb12u2_amd64.deb (--unpack):
 unable to make backup link of './usr/bin/perl' before installing new version: Operation not permitted
Errors were encountered while processing:
 /var/cache/apt/archives/perl-base_5.36.0-7+deb12u2_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@681b04feebbe:/#

Here is sysbox-runc version that I use:

sysbox-runc --version
sysbox-runc
        edition:        Community Edition (CE)
        version:        0.6.3
        commit:         e6ca9b930c43c0f20bb93ef3ef6af5bd49fd88ce
        built at:       Tue Jan  9 17:59:49 UTC 2024
        built by:       Cesar Talledo
        oci-specs:      1.1.0+dev

I think this might be the same error due to the following error and the mention of the perl update by the original poster:

dpkg: error processing archive /var/cache/apt/archives/perl-base_5.36.0-7+deb12u2_amd64.deb (--unpack):
 unable to make backup link of './usr/bin/perl' before installing new version: Operation not permitted

I can also reproduce this more directly:

docker run --rm -it debian:bookworm /bin/bash
root@64acc8d53500:/# ln /usr/bin/perl /usr/bin/perl.bak
ln: failed to create hard link '/usr/bin/perl.bak' => '/usr/bin/perl': Operation not permitted

[smarsching]

@ctalledo Thanks for trying to reproduce the problem. I checked, and the problem does indeed not exist when using the HWE (6.8) kernel. It can only be reproduced when using the “regular” (5.15) kernel for this Ubuntu release.

root@ubuntu:~# uname -a
Linux tuxy 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Depending on how and when Ubuntu was installed, this is the default kernel. It can also be installed manually through the the linux-image-generic package. I specifically tested with linux-image-5.15.0-136-generic.

Is it expected that recent versions of Sysbox do not work with the 5.15 kernel?

Here is my Docker version info:

root@ubuntu:~# docker version
Client:
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.22.2
 Git commit:        26.1.3-0ubuntu1~22.04.1
 Built:             Mon Oct 14 21:24:40 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.22.2
  Git commit:       26.1.3-0ubuntu1~22.04.1
  Built:            Mon Oct 14 21:24:40 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        
 sysbox-runc:
  Version:          :                   0.6.6
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

@href I do not know why you also see this problem with Sysbox 0.6.3. I installed this version in a barely touched Ubuntu 22.04 system (basically a VM clone, where only virtualization utilities and system package updates had previously been installed), and there I could only reproduce the bug with version 0.6.6, not with version 0.6.3. So, this might indeed indicate whether you are experiencing a different bug, that only has the same symptoms. Have you tried to reproduce this on a freshly installed system in order to rule out that it is related to some configuration that might be specific to your system?

[href]

It's fairly possible that this has something to do with our setup. We still run this on 20.04. Since you cannot reproduce it on 22.04, and because we have to update anyway, I'll go ahead and update first, and then post here again (or open another issue, as this might take us a moment).

[ctalledo]

@href: on my Ubuntu-Jammy host with kernel 6.8, I am still not able to repro with:

docker run --rm -it debian@sha256:00cd074b40c4d99ff0c24540bdde0533ca3791edcdac0de36d6b9fb3260d89e2

@smarsching:

it can only be reproduced when using the “regular” (5.15) kernel for this Ubuntu release.

Ah that's interesting; a few more questions:

• What version of Ubuntu are you running?
• Did the Sysbox installer load the shiftfs module? (lsmod | grep shiftfs)
• Inside the debian container, please post the output of findmnt.

[smarsching]

@ctalledo

I am running Ubuntu 22.04 and yes, shiftfs is loaded (I guess it would affect all files and not just files with more than one hard link if it was a general problem with shiftfs).

lsmod | grep shiftfs:

shiftfs                28672  0

lsb_release -a:

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy

findmnt inside the container:

TARGET             SOURCE                                         FSTYPE OPTIONS
/                  overlay                                        overla rw,rela
|-/sys             sysfs                                          sysfs  rw,nosu
| |-/sys/firmware  tmpfs                                          tmpfs  ro,rela
| |-/sys/fs/cgroup cgroup                                         cgroup rw,nosu
| |-/sys/devices/virtual
| |                sysboxfs[/sys/devices/virtual]                 fuse   rw,nosu
| | `-/sys/devices/virtual/powercap
| |                tmpfs                                          tmpfs  ro,rela
| |-/sys/kernel    sysboxfs[/sys/kernel]                          fuse   rw,nosu
| `-/sys/module/nf_conntrack/parameters
|                  sysboxfs[/sys/module/nf_conntrack/parameters]  fuse   rw,nosu
|-/proc            proc                                           proc   rw,nosu
| |-/proc/bus      proc[/bus]                                     proc   ro,nosu
| |-/proc/fs       proc[/fs]                                      proc   ro,nosu
| |-/proc/irq      proc[/irq]                                     proc   ro,nosu
| |-/proc/sysrq-trigger
| |                proc[/sysrq-trigger]                           proc   ro,nosu
| |-/proc/asound   tmpfs                                          tmpfs  ro,rela
| |-/proc/acpi     tmpfs                                          tmpfs  ro,rela
| |-/proc/keys     udev[/null]                                    devtmp rw,nosu
| |-/proc/timer_list
| |                udev[/null]                                    devtmp rw,nosu
| |-/proc/scsi     tmpfs                                          tmpfs  ro,rela
| |-/proc/swaps    sysboxfs[/proc/swaps]                          fuse   rw,nosu
| |-/proc/sys      sysboxfs[/proc/sys]                            fuse   rw,nosu
| `-/proc/uptime   sysboxfs[/proc/uptime]                         fuse   rw,nosu
|-/dev             tmpfs                                          tmpfs  rw,nosu
| |-/dev/console   devpts[/0]                                     devpts rw,nosu
| |-/dev/mqueue    mqueue                                         mqueue rw,nosu
| |-/dev/pts       devpts                                         devpts rw,nosu
| |-/dev/shm       shm                                            tmpfs  rw,nosu
| |-/dev/null      udev[/null]                                    devtmp rw,nosu
| |-/dev/random    udev[/random]                                  devtmp rw,nosu
| |-/dev/kmsg      udev[/null]                                    devtmp rw,nosu
| |-/dev/full      udev[/full]                                    devtmp rw,nosu
| |-/dev/tty       udev[/tty]                                     devtmp rw,nosu
| |-/dev/zero      udev[/zero]                                    devtmp rw,nosu
| `-/dev/urandom   udev[/urandom]                                 devtmp rw,nosu
|-/var/lib/k0s     /dev/mapper/vg0-root[/var/lib/sysbox/k0s/71a3bd38a8f5a9ac0591d4a0437d6cc2934e6e2880ab3154ad822f72c7b70161]
|                                                                 ext4   rw,rela
|-/etc/resolv.conf /dev/mapper/vg0-root[/var/lib/docker/containers/71a3bd38a8f5a9ac0591d4a0437d6cc2934e6e2880ab3154ad822f72c7b70161/resolv.conf]
|                                                                 ext4   rw,rela
|-/etc/hostname    /dev/mapper/vg0-root[/var/lib/docker/containers/71a3bd38a8f5a9ac0591d4a0437d6cc2934e6e2880ab3154ad822f72c7b70161/hostname]
|                                                                 ext4   rw,rela
|-/etc/hosts       /dev/mapper/vg0-root[/var/lib/docker/containers/71a3bd38a8f5a9ac0591d4a0437d6cc2934e6e2880ab3154ad822f72c7b70161/hosts]
|                                                                 ext4   rw,rela
|-/var/lib/buildkit
|                  /dev/mapper/vg0-root[/var/lib/sysbox/buildkit/71a3bd38a8f5a9ac0591d4a0437d6cc2934e6e2880ab3154ad822f72c7b70161]
|                                                                 ext4   rw,rela
|-/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
|                  /dev/mapper/vg0-root[/var/lib/sysbox/containerd/71a3bd38a8f5a9ac0591d4a0437d6cc2934e6e2880ab3154ad822f72c7b70161]
|                                                                 ext4   rw,rela
|-/var/lib/docker  /dev/mapper/vg0-root[/var/lib/sysbox/docker/71a3bd38a8f5a9ac0591d4a0437d6cc2934e6e2880ab3154ad822f72c7b70161]
|                                                                 ext4   rw,rela
|-/var/lib/rancher/k3s
|                  /dev/mapper/vg0-root[/var/lib/sysbox/rancher-k3s/71a3bd38a8f5a9ac0591d4a0437d6cc2934e6e2880ab3154ad822f72c7b70161]
|                                                                 ext4   rw,rela
|-/var/lib/rancher/rke2
|                  /dev/mapper/vg0-root[/var/lib/sysbox/rancher-rke2/71a3bd38a8f5a9ac0591d4a0437d6cc2934e6e2880ab3154ad822f72c7b70161]
|                                                                 ext4   rw,rela
|-/var/lib/kubelet /dev/mapper/vg0-root[/var/lib/sysbox/kubelet/71a3bd38a8f5a9ac0591d4a0437d6cc2934e6e2880ab3154ad822f72c7b70161]
|                                                                 ext4   rw,rela
|-/usr/src/linux-headers-5.15.0-136
|                  /dev/mapper/vg0-root[/usr/src/linux-headers-5.15.0-136]
|                                                                 ext4   ro,rela
|-/usr/src/linux-headers-5.15.0-136-generic
|                  /dev/mapper/vg0-root[/usr/src/linux-headers-5.15.0-136-generic]
|                                                                 ext4   ro,rela
`-/usr/lib/modules/5.15.0-136-generic
                   /dev/mapper/vg0-root[/usr/lib/modules/5.15.0-136-generic]
                                                                  ext4   ro,rela

I still find it curious that inside the container, the two paths reference different inodes, while outside the container they reference the same inode.

stat /var/lib/docker/overlay2/c24b43df210e9decd58191d1737e6b1f20a50b1142d4c0849611b21d94270293/diff/usr/bin/perl* outside the container:

  File: /var/lib/docker/overlay2/c24b43df210e9decd58191d1737e6b1f20a50b1142d4c0849611b21d94270293/diff/usr/bin/perl
  Size: 3804432   	Blocks: 7432       IO Block: 4096   regular file
Device: fd01h/64769d	Inode: 1054966     Links: 2
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-11-25 21:59:54.000000000 +0100
Modify: 2023-11-25 21:59:54.000000000 +0100
Change: 2025-04-19 13:18:12.495608141 +0200
 Birth: 2025-04-19 13:18:12.479608415 +0200
  File: /var/lib/docker/overlay2/c24b43df210e9decd58191d1737e6b1f20a50b1142d4c0849611b21d94270293/diff/usr/bin/perl5.36.0
  Size: 3804432   	Blocks: 7432       IO Block: 4096   regular file
Device: fd01h/64769d	Inode: 1054966     Links: 2
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-11-25 21:59:54.000000000 +0100
Modify: 2023-11-25 21:59:54.000000000 +0100
Change: 2025-04-19 13:18:12.495608141 +0200
 Birth: 2025-04-19 13:18:12.479608415 +0200

stat /usr/bin/perl* inside the container:

  File: /usr/bin/perl
  Size: 3804432   	Blocks: 7432       IO Block: 4096   regular file
Device: 0,62	Inode: 1054966     Links: 2
Access: (0755/-rwxr-xr-x)  Uid: (65534/  nobody)   Gid: (65534/ nogroup)
Access: 2023-11-25 20:59:54.000000000 +0000
Modify: 2023-11-25 20:59:54.000000000 +0000
Change: 2025-04-19 11:18:12.495608141 +0000
 Birth: 2025-04-19 11:18:12.479608415 +0000
  File: /usr/bin/perl5.36.0
  Size: 3804432   	Blocks: 7432       IO Block: 4096   regular file
Device: 0,62	Inode: 1086533     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-11-25 20:59:54.000000000 +0000
Modify: 2023-11-25 20:59:54.000000000 +0000
Change: 2025-04-19 11:18:14.859567605 +0000
 Birth: 2025-04-19 11:18:14.859567605 +0000

I suspect that this is somehow related to the ownership problem. When using the regular runc runtime instead of sysbox-runc, both paths inside the container refer to the same inode (1054966) and both have the correct owner (root:root).

[ctalledo]

Hi @smarsching,

Unfortunately the output of findmnt is cut-off on the right side, I need the full output to check how the container's rootfs is setup. Please attach a file with the full output.

Thanks.

[smarsching]

@ctalledo You are right. Because of the long paths that were not truncated, I missed that the mount options were. Here is the full output:

TARGET                                                       SOURCE                                                                                                                        FSTYPE   OPTIONS
/                                                            overlay                                                                                                                       overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/XNHREDILMU7Q6TJU3LQVDALW4F:/var/lib/docker/overlay2/l/S57GST4ALVQYWRFHRPAIBC6CFK,upperdir=/var/lib/sysbox/rootfs/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4/overlay2/diff,workdir=/var/lib/sysbox/rootfs/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4/overlay2/work,metacopy=on
|-/sys                                                       sysfs                                                                                                                         sysfs    rw,nosuid,nodev,noexec,relatime
| |-/sys/firmware                                            tmpfs                                                                                                                         tmpfs    ro,relatime,uid=362144,gid=362144,inode64
| |-/sys/fs/cgroup                                           cgroup                                                                                                                        cgroup2  rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
| |-/sys/devices/virtual                                     sysboxfs[/sys/devices/virtual]                                                                                                fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| | `-/sys/devices/virtual/powercap                          tmpfs                                                                                                                         tmpfs    ro,relatime,uid=362144,gid=362144,inode64
| |-/sys/kernel                                              sysboxfs[/sys/kernel]                                                                                                         fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| `-/sys/module/nf_conntrack/parameters                      sysboxfs[/sys/module/nf_conntrack/parameters]                                                                                 fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
|-/proc                                                      proc                                                                                                                          proc     rw,nosuid,nodev,noexec,relatime
| |-/proc/bus                                                proc[/bus]                                                                                                                    proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/fs                                                 proc[/fs]                                                                                                                     proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/irq                                                proc[/irq]                                                                                                                    proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/sysrq-trigger                                      proc[/sysrq-trigger]                                                                                                          proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/asound                                             tmpfs                                                                                                                         tmpfs    ro,relatime,uid=362144,gid=362144,inode64
| |-/proc/acpi                                               tmpfs                                                                                                                         tmpfs    ro,relatime,uid=362144,gid=362144,inode64
| |-/proc/keys                                               udev[/null]                                                                                                                   devtmpfs rw,nosuid,relatime,size=16373984k,nr_inodes=4093496,mode=755,inode64
| |-/proc/timer_list                                         udev[/null]                                                                                                                   devtmpfs rw,nosuid,relatime,size=16373984k,nr_inodes=4093496,mode=755,inode64
| |-/proc/scsi                                               tmpfs                                                                                                                         tmpfs    ro,relatime,uid=362144,gid=362144,inode64
| |-/proc/swaps                                              sysboxfs[/proc/swaps]                                                                                                         fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| |-/proc/sys                                                sysboxfs[/proc/sys]                                                                                                           fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| `-/proc/uptime                                             sysboxfs[/proc/uptime]                                                                                                        fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
|-/dev                                                       tmpfs                                                                                                                         tmpfs    rw,nosuid,size=65536k,mode=755,uid=362144,gid=362144,inode64
| |-/dev/console                                             devpts[/0]                                                                                                                    devpts   rw,nosuid,noexec,relatime,gid=362149,mode=620,ptmxmode=666
| |-/dev/mqueue                                              mqueue                                                                                                                        mqueue   rw,nosuid,nodev,noexec,relatime
| |-/dev/pts                                                 devpts                                                                                                                        devpts   rw,nosuid,noexec,relatime,gid=362149,mode=620,ptmxmode=666
| |-/dev/shm                                                 shm                                                                                                                           tmpfs    rw,nosuid,nodev,noexec,relatime,size=65536k,uid=362144,gid=362144,inode64
| |-/dev/null                                                udev[/null]                                                                                                                   devtmpfs rw,nosuid,relatime,size=16373984k,nr_inodes=4093496,mode=755,inode64
| |-/dev/random                                              udev[/random]                                                                                                                 devtmpfs rw,nosuid,relatime,size=16373984k,nr_inodes=4093496,mode=755,inode64
| |-/dev/kmsg                                                udev[/null]                                                                                                                   devtmpfs rw,nosuid,relatime,size=16373984k,nr_inodes=4093496,mode=755,inode64
| |-/dev/full                                                udev[/full]                                                                                                                   devtmpfs rw,nosuid,relatime,size=16373984k,nr_inodes=4093496,mode=755,inode64
| |-/dev/tty                                                 udev[/tty]                                                                                                                    devtmpfs rw,nosuid,relatime,size=16373984k,nr_inodes=4093496,mode=755,inode64
| |-/dev/zero                                                udev[/zero]                                                                                                                   devtmpfs rw,nosuid,relatime,size=16373984k,nr_inodes=4093496,mode=755,inode64
| `-/dev/urandom                                             udev[/urandom]                                                                                                                devtmpfs rw,nosuid,relatime,size=16373984k,nr_inodes=4093496,mode=755,inode64
|-/var/lib/docker                                            /dev/mapper/vg0-root[/var/lib/sysbox/docker/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4]                 ext4     rw,relatime,errors=remount-ro
|-/etc/resolv.conf                                           /dev/mapper/vg0-root[/var/lib/docker/containers/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4/resolv.conf] ext4     rw,relatime,idmapped,errors=remount-ro
|-/etc/hostname                                              /dev/mapper/vg0-root[/var/lib/docker/containers/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4/hostname]    ext4     rw,relatime,idmapped,errors=remount-ro
|-/etc/hosts                                                 /dev/mapper/vg0-root[/var/lib/docker/containers/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4/hosts]       ext4     rw,relatime,idmapped,errors=remount-ro
|-/var/lib/rancher/k3s                                       /dev/mapper/vg0-root[/var/lib/sysbox/rancher-k3s/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4]            ext4     rw,relatime,errors=remount-ro
|-/var/lib/rancher/rke2                                      /dev/mapper/vg0-root[/var/lib/sysbox/rancher-rke2/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4]           ext4     rw,relatime,errors=remount-ro
|-/var/lib/kubelet                                           /dev/mapper/vg0-root[/var/lib/sysbox/kubelet/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4]                ext4     rw,relatime,errors=remount-ro
|-/var/lib/k0s                                               /dev/mapper/vg0-root[/var/lib/sysbox/k0s/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4]                    ext4     rw,relatime,errors=remount-ro
|-/var/lib/buildkit                                          /dev/mapper/vg0-root[/var/lib/sysbox/buildkit/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4]               ext4     rw,relatime,errors=remount-ro
|-/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs /dev/mapper/vg0-root[/var/lib/sysbox/containerd/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4]             ext4     rw,relatime,errors=remount-ro
|-/usr/src/linux-headers-5.15.0-136                          /dev/mapper/vg0-root[/usr/src/linux-headers-5.15.0-136]                                                                       ext4     ro,relatime,idmapped,errors=remount-ro
|-/usr/src/linux-headers-5.15.0-136-generic                  /dev/mapper/vg0-root[/usr/src/linux-headers-5.15.0-136-generic]                                                               ext4     ro,relatime,idmapped,errors=remount-ro
`-/usr/lib/modules/5.15.0-136-generic                        /dev/mapper/vg0-root[/usr/lib/modules/5.15.0-136-generic]                                                                     ext4     ro,relatime,idmapped,errors=remount-ro

[ctalledo]

Hi @smarsching , thanks for pasting the full findmnt output.

I can see that Sysbox is not using shiftfs, but rather using the less desirable "rootfs-cloning" approach.

/          overlay        rw,relatime,lowerdir= ....,upperdir=/var/lib/sysbox/rootfs/cc6e0ccbbe3baf80d8a78d661fab11b1f5f48e136f8e846880d244faa444fec4/overlay2/diff,...

Since you indicated shiftfs is installed, it must be that Sysbox detected that shiftfs is not working correctly (it checks for this when it starts).

Can you do the following to enable debug logging in the sysbox-mgr?

sudo sed -i 's@ExecStart=/usr/bin/sysbox-mgr@ExecStart=/usr/bin/sysbox-mgr --log-level=debug@' /lib/systemd/system/sysbox-mgr.service
sudo systemctl daemon-reload

Then start the sysbox-mgr and show me the logs:

sudo systemctl restart sysbox-mgr
journalctl -u sysbox-mgr

For your machine with kernel 5.15, it should show something like this:

May 02 11:11:06 lenovo sysbox-mgr[2857231]: time="2025-05-02 11:11:06" level=info msg="Shiftfs module found in kernel: yes"
May 02 11:11:06 lenovo sysbox-mgr[2857231]: time="2025-05-02 11:11:06" level=info msg="Shiftfs works properly: yes"
May 02 11:11:06 lenovo sysbox-mgr[2857231]: time="2025-05-02 11:11:06" level=info msg="Shiftfs-on-overlayfs works properly: no"

Having said this, I need to understand why the rootfs-cloning approach is not setting up the ownership correctly for the files you mentioned. I suspect it's because of your earlier observation:

I am not entirely sure why the file /usr/bin/perl5.38.2 (and /usr/bin/gunzip). I suspect that the problem is related to the fact that these are hard links.

But rootfs-cloning was a temporary solution (neither it nor shiftfs are needed with kernel 5.19+, as they are replaced with a kernel mechanism called idmapped mounts), so won't spent much time on it.

[smarsching]

@ctalledo You are right, there is a problem with shifts:

May 02 23:21:03 tuxy sysbox-mgr[340944]: time="2025-05-02 23:21:03" level=info msg="Shiftfs module found in kernel: yes"
May 02 23:21:03 tuxy sysbox-mgr[340944]: time="2025-05-02 23:21:03" level=info msg="Shiftfs works properly: no"
May 02 23:21:03 tuxy sysbox-mgr[340944]: time="2025-05-02 23:21:03" level=info msg="Shiftfs-on-overlayfs works properly: yes"

Do you have any suggestion how I might investigate why shiftfs does not work properly?

[ctalledo]

Thanks @smarsching for confirming.

Do you have any suggestion how I might investigate why shiftfs does not work properly?

It could be that shiftfs is not working properly in your kernel, or a bug in the sysbox-mgr code that checks if shiftfs is working or not.

Could you please enable debug logging in the sysbox-mgr and restart Sysbox? (see my prior comment on how to do this). It will then report more info on why it thinks shiftfs is not working.

[ctalledo]

It could be that shiftfs is not working properly in your kernel, or a bug in the sysbox-mgr code that checks if shiftfs is working or not.

I am pretty sure it's the latter (a bug in sysbox-mgr), because I was able to reproduce on a Linux host with 5.15 kernel yet confirm that shiftfs is actually working properly (even though sysbox-mgr thinks it's not).

I'll investigate and fix.

Work-around:

Disable the "shiftfs precheck" in the sysbox-mgr, by editing the /lib/systemd/system/sysbox-mgr.service file and adding the following flag to the ExecStart line:

ExecStart=/usr/bin/sysbox-mgr --disable-shiftfs-precheck

Then:

sudo systemctl daemon-reload
sudo systemctl restart sysbox

And then starting a container with docker run --runtime=sysbox-runc -it --rm alpine and verifying that findmnt shows that shiftfs is mounted inside the container's rootfs:

/            .           shiftfs rw,relatime