Buster builds broken on non-arm hosts

[XECDesign]

Opening this as a heads to anyone relying on pi-gen.

https://bugs.launchpad.net/qemu/+bug/1805913

Unless this bug is fixed by the time buster goes live, images built through qemu-arm-static are going to be broken in slightly subtle ways. Luckily, qemu devs are pretty good and the issue is likely to be resolved before then.

pixbuf relies on the mime database, which silently fails to update and returns success. The result is that desktop is rendered without any icons.

Something similar happens with SSL certificates, breaking rpi-update and anything else that wants to use https.

Those are the known ways images break, but any binary that uses readdir() is not going to work.

Internally, we've moved our builds to an arm build server to avoid going through qemu for now.

[hhromic]

@XECDesign thanks for the heads-up. And congratulations on releasing the new RPI4 with Buster!

Are you using this same pi-gen repository internally with your arm build servers? In other words is this repository still considered the official reference Raspbian image builder? Are you planning to update pi-gen to build buster images regardless of the problem with non-arm hosts? Thanks!

[XECDesign]

I've just pushed the commits that we were using internally, but couldn't make public yet.

Not sure how to approach non-arm host builds right now.

[hhromic]

Thanks a lot for that! Appreciated!
I guess for now we have to just wait for the qemu issue to get fixed upstream, unfortunately, i.e. patience.

[XECDesign]

It's looking more like a kernel issue, but discussions I've seen on the mailing list seem to have fizzled out a long time ago without any resolution. Maybe when Buster is more commonly used it will press the issue.

[hhromic]

Hopefully it gets more attention now that Buster is about to go live.
One final question in case you know from the top of your head: this problem only happens when the host is 64 bits or it happens on any host that is non-arm? If the host is non-arm but 32 bits do you think it would work ok? I can test using a VM if you are not sure.

[XECDesign]

Sorry, not sure off the top of my head.

[hhromic]

No problem, will quickly setup a VM with 32-bits Debian and check it out, if it works, then we have a temporary solution for the moment until it's fixed. Will report back. Thanks again!

[hhromic]

Hi @XECDesign , so I can confirm now that this issue is specifically for hosts with 64 bits kernels no matter if arm or not. I made a testing VM with vanilla Debian i386 (32 bits kernel) and the generated image works fine on real hardware (tested with RPI 3B).

To verify this I built a control "broken" Buster image with a 64 bits Debian host using Docker and I did get SSL problems with curl (with GitHub and other websites too), for example:

$ curl -sSL https://github.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Then I built another Buster image using a 32 bits Debian host and the curl command above worked fine on the same hardware and same network almost at the same time.

Aside, I also noticed a minor issue with the Qemu version shipped with Debian Stretch, where the man-db package being installed for Buster in the image triggers many of these errors:

qemu: Unsupported syscall: 383

This is a manifestation of the following bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891109

Fortunately this is resolved in the current Qemu version shipped with Debian Buster, therefore building with a Debian Buster host will not show any errors. I will send a PR to update the Dockerfile for this.

In summary, I wanted to let you know that this bug is not affecting 32 bits build hosts, no matter if they are arm or not (at least for me). For now you can use a 32 bits build host and pi-gen will generate a working image.

Hope this is useful for future readers!

[XECDesign]

Thanks for looking into it. Much appreciated.

[ali1234]

On a similar note, if you run buster in a chroot on Ubuntu 18.04 you will need to upgrade proot to 5.1.0-1.13 (if you use it) and qemu-user to something newer than 2.11 (3.1 works). This is because buster uses renameat2() and new features of getauxval(). These versions are available in Ubuntu 19.04 but not 18.04 LTS. In particular bootstrapping will not work in proot without both these upgrades because 'mv' command will not work, which completely borks the preinst and postinst scripts.

This bug is an example of the kind of issue you'll see with too old qemu versions:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923289

[Chaser]

Hi all, we were able to overcome the issue of SSL certs only by rehashing the ssl certs with c_rehash

Specifically

# Patch Issue with ssl certs as per https://github.com/RPi-Distro/pi-gen/issues/271
on_chroot << EOF
echo Patching certs...
c_rehash /etc/ssl/certs
EOF

Credit to Keith Tate as well :)

As indicated below by @hhromic its not a complete solution.

[hhromic]

@Chaser that is a workaround for the ca-certificates package only, not a solution to the actual problem described in this issue. I wouldn't advice advertising it as a solution.

The problem affects any program using the readdir() syscall, not just ca-certificates and the effects are of varying nature. The SSL certificates issue is just one manifestation. Another (as the original post indicated) are icons not being correctly rendered in the Desktop.

It is unknown/unverified what other packages might be affected. Therefore the safest solution for now is to build using a 32-bits host (be ARM or non-ARM) as indicated before.