qemu-multiarch support

[mhornbacher]

Multi-arch buildx builds currently do not work on the sysbox runtime due to lack of support for this feature.

As per this slack message it is actively being worked on and this ticket is to track progress.

[ctalledo]

@rodnymolina in case he has further info on where we are at with running multiarch builds with Sysbox.

[rodnymolina]

To move forward with this, we first need to allow containerized processes to write into the /proc/sys/fs/binfmt_misc node in order to enable the different qemu arch-specific binaries to be invoked whenever required. And we need to do this securely.

Unfortunately, this isn't a trivial task, since this binfmt_misc node is a non-namespaced system-wide resource, so if we were to allow changes to it within a sysbox container, this could potentially open an attack vector on the overall system. That's just to say that we need to be careful here.

Notice that the solutions that currently exist to address this issue at the host level are not applicable to our case since they require the execution of --privileged containers to grant full access to the binfmt_misc node (something that we're trying to avoid as explained above).

[h4l]

I've found I can do multi-arch buildx builds on a dockerd running inside an unprivileged sysbox-runc container as long as the underlying OS where sysbox-runc is installed already has the qemu binaries registered under /proc/sys/fs/binfmt_misc/. Is this the expected behaviour?

If so, is the scope of this issue limited to just modifying or registering new binfmt_misc entries? In that case, I'd suggest we update the limitations page to clarify this.

[mhornbacher]

Interesting, could you elaborate on how you installed qemu binaries there?

[h4l]

Sure. This is a bit long, so I'll collapse it!

Details

(I found this post useful, I was kind of freestyling with ideas from it: https://medium.com/@artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408 .)

The top-level host OS is Ubuntu 20.04. I installed the qemu-user-static package there (along with binfmt-support which is a recommended package for it). This gives me various architectures registered:

$ ls /proc/sys/fs/binfmt_misc/
python3.8     qemu-alpha  qemu-armeb  qemu-hppa  qemu-microblaze  qemu-mips64    qemu-mipsel   qemu-mipsn32el  qemu-ppc64       qemu-ppc64le  qemu-riscv64  qemu-sh4    qemu-sparc        qemu-sparc64  qemu-xtensaeb  status
qemu-aarch64  qemu-arm    qemu-cris   qemu-m68k  qemu-mips        qemu-mips64el  qemu-mipsn32  qemu-ppc        qemu-ppc64abi32  qemu-riscv32  qemu-s390x    qemu-sh4eb  qemu-sparc32plus  qemu-xtensa   register
$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: OCF
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

I've got sysbox-runc 0.5.2 installed too, and dockerd 20.10.14.

I've run buildkitd under sysbox-runc successfully in two ways. I'm setting up buildkitd to use with a GitLab CI runner. I'd initially tried and failed to do multiarch builds when running dockerd inside the GitLab CI Runner's container (which is how I found this issue). So after that I had been planning to set up a standalone buildkitd and share it with the runners as a remote builder, so setting that up is how I noticed that it actually works under sysbox-runc (after installing the qemu binaries on the top-level host OS):

$ docker network create buildkit-test
$ docker container run --rm --runtime sysbox-runc  --name=remote-buildkitd --network buildkit-test moby/buildkit:latest --addr tcp://0.0.0.0:1234
...

So this container seems to inherit the same binfmt registrations as the host:

$ docker container exec -it remote-buildkitd sh
/ # ls /proc/sys/fs/binfmt_misc/
python3.8         qemu-arm          qemu-hppa         qemu-mips         qemu-mipsel       qemu-ppc          qemu-ppc64le      qemu-s390x        qemu-sparc        qemu-xtensa       status
qemu-aarch64      qemu-armeb        qemu-m68k         qemu-mips64       qemu-mipsn32      qemu-ppc64        qemu-riscv32      qemu-sh4          qemu-sparc32plus  qemu-xtensaeb
qemu-alpha        qemu-cris         qemu-microblaze   qemu-mips64el     qemu-mipsn32el    qemu-ppc64abi32   qemu-riscv64      qemu-sh4eb        qemu-sparc64      register
/ # cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: OCF
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

The actual binaries don't exist in the container though:

/ # stat /usr/bin/qemu-aarch64-static
stat: can't stat '/usr/bin/qemu-aarch64-static': No such file or directory

Then I can run another container with docker CLI in that network and connect it to this builder as a remote builder:

$ docker container run --rm -it --network buildkit-test docker
/ # docker buildx create --name remote-buildkitd --driver remote tcp://remote-buildkitd:1234
remote-buildkitd
/ # docker buildx ls
NAME/NODE           DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
remote-buildkitd    remote
  remote-buildkitd0 tcp://remote-buildkitd:1234 running          linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default *                                       error

Cannot load builder default *: error during connect: Get "http://docker:2375/_ping": dial tcp: lookup docker on 172.25.3.1:53: server misbehaving

And I just made a simple Dockerfile to do a test build, I think it was just:

FROM alpine
RUN echo "Hi from $(uname -a)"

And this builds successfully in arm64 and amd64:

/tmp/docker # docker buildx use remote-buildkitd
/tmp/docker # ls
Dockerfile
/tmp/docker # docker buildx build --platform linux/arm64,linux/amd64 --progress=plain .
WARNING: No output specified with remote driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
#1 [internal] load .dockerignore
#1 transferring context: 2B 0.0s done
#1 DONE 0.1s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 88B 0.0s done
#2 DONE 0.1s

#3 [linux/arm64 internal] load metadata for docker.io/library/alpine:latest
#3 DONE 2.5s

#4 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
#4 DONE 2.5s

#5 [linux/arm64 1/2] FROM docker.io/library/alpine:latest@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4
#5 resolve docker.io/library/alpine:latest@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 0.0s done
#5 sha256:261da4162673b93e5c0e7700a3718d40bcc086dbf24b1ec9b54bca0b82300626 3.26MB / 3.26MB 0.2s done
#5 extracting sha256:261da4162673b93e5c0e7700a3718d40bcc086dbf24b1ec9b54bca0b82300626
#5 extracting sha256:261da4162673b93e5c0e7700a3718d40bcc086dbf24b1ec9b54bca0b82300626 0.2s done
#5 DONE 0.4s

#6 [linux/amd64 1/2] FROM docker.io/library/alpine:latest@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4
#6 resolve docker.io/library/alpine:latest@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 0.0s done
#6 sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 3.37MB / 3.37MB 0.3s done
#6 extracting sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715
#6 extracting sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 0.1s done
#6 DONE 0.4s

#7 [linux/amd64 2/2] RUN echo "Hi from $(uname -a)"
#0 0.070 Hi from Linux buildkitsandbox 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 Linux
#7 DONE 0.2s

#8 [linux/arm64 2/2] RUN echo "Hi from $(uname -a)"
#0 0.135 Hi from Linux buildkitsandbox 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 aarch64 Linux
#8 DONE 0.2s

So that's fine. But with the qemu binaries installed on the host, I'm able to just start a buildkit builder on a dockerd running inside a CI build container. The CI runner uses the Docker executor. My config for the Docker executor is this:

# ...
  [runners.docker]
    tls_verify = true
    image = "ubuntu:20.04"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache", "/var/lib/docker"]
    shm_size = 0
    runtime = "sysbox-runc"
    extra_hosts = ["gitlab-ci-minio:host-gateway"]

(I guess I probably shouldn't share /var/lib/docker with the runners...). My CI job that runs my buildx build runs on an image I created that's basically https://github.com/nestybox/dockerfiles/tree/master/alpine-supervisord-docker — it contains its own dockerd. In the CI job all I do is

$ docker buildx create --use
$ docker buildx bake --push

So buildkitd is running under the dockerd running in the CI job's container. And this builds a real image with plenty of stuff in, not just the toy Dockerfile above.

[mhornbacher]

Thank you @h4l!

With some minor modifications we got it working in our Buildkite environment as well!

All our modifications where tweaks around using an existing per-job network as well as loading up the buildx configuration in every builds environment hooks :)

[h4l]

@mhornbacher Nice job, that's great, glad you got it working!

[mhornbacher]

@rodnymolina @ctalledo I think this is a fairly workable solution for most customers. Big shoutout to @h4l for his discoveries!

For anyone stumbling upon this via a search.

1. Install qemu-user-static and binfmt-support which fixes running moby/buildkit under sysbox.
2. Expose moby/buildkit to your container via the docker network (either via a custom bridge network or the default one) and use it as a remote builder

Implementation details are in the above two comments 👍

Feel free to @ me with a comment here if you need more explanation

[ctalledo]

Thanks @mhornbacher for your contribution to resolving this issue, much appreciated!

[rodnymolina]

Just a heads-up that the workaround mentioned above to allow multi-arch builds within a sysbox container is not working as of v0.6.2 release (thanks to @DekusDenial for letting us know).

This is a consequence of a recent change to ensure that Sysbox exposes all the procfs and sysfs nodes known within the container's namespaces. As a side-effect of this, we stopped exposing a few host nodes within sysbox containers, which on one hand offers greater security, but on the other, it breaks functionality like the one required by the above workaround.

As a fix, we could make an exception for these nodes by creating a new 'handler' in sysbox-fs to expose the host's /proc/sys/fs/binfmt_misc nodes. We will take this into account for the next release (ping us if you can't wait).

[mhornbacher]

Thanks. Pinging @jmandel1027 :)

[johnwmail]

I have same problem to run CICD, any workaround? thanks.

[mhornbacher]

You can use version 0.6.1 with the workarounds detailed above if you need mutli arch builds for now. This is no longer my day-to-day so I am not on top of any 0.6.2 workarounds yet