Stale servers

[yosifkit]

Some servers in the list work fine, others are unresponsive ("connection timed out" or "no data"), and others have an outdated key (even a month later 😞). These outdated keyservers are sometimes fastest and so the client gets served the outdated key.

When there were issues back on January 24 with ros image builds in GitHub actions, I had assumed it was just gossip lag. But it is still an issue on their latest PR.

$ # choosing a set of servers that happen to display all states
root@690deddaab77:/# for serv in keyserver.maxweiss.io keyserver.snt.utwente.nl keyserver.spline.inf.fu-berlin.de keys.i2p-projekt.de; do export GNUPGHOME="$(mktemp -d)"; echo "$serv":; gpg --batch --keyserver "hkp://$serv" --recv-keys 'C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654'; gpg --fingerprint; gpgconf --kill all; sleep 1; rm -rf "$GNUPGHOME"; done
keyserver.maxweiss.io:
gpg: keybox '/tmp/tmp.ptqzBy0GKt/pubring.kbx' created
gpg: /tmp/tmp.ptqzBy0GKt/trustdb.gpg: trustdb created
gpg: key F42ED6FBAB17C654: public key "Open Robotics <info@osrfoundation.org>" imported
gpg: Total number processed: 1
gpg:               imported: 1
/tmp/tmp.ptqzBy0GKt/pubring.kbx
-------------------------------
pub   rsa4096 2019-05-30 [SC] [expires: 2025-06-01]
      C1CF 6E31 E6BA DE88 68B1  72B4 F42E D6FB AB17 C654
uid           [ unknown] Open Robotics <info@osrfoundation.org>

keyserver.snt.utwente.nl:
gpg: keybox '/tmp/tmp.0pFeCCtLjq/pubring.kbx' created
gpg: keyserver receive failed: Connection timed out
gpg: /tmp/tmp.0pFeCCtLjq/trustdb.gpg: trustdb created
keyserver.spline.inf.fu-berlin.de:
gpg: keybox '/tmp/tmp.PRub9GpThs/pubring.kbx' created
gpg: keyserver receive failed: No data
gpg: /tmp/tmp.PRub9GpThs/trustdb.gpg: trustdb created
keys.i2p-projekt.de:
gpg: keybox '/tmp/tmp.zVYOoe9F0Q/pubring.kbx' created
gpg: /tmp/tmp.zVYOoe9F0Q/trustdb.gpg: trustdb created
gpg: key F42ED6FBAB17C654: public key "Open Robotics <info@osrfoundation.org>" imported
gpg: Total number processed: 1
gpg:               imported: 1
/tmp/tmp.zVYOoe9F0Q/pubring.kbx
-------------------------------
pub   rsa4096 2019-05-30 [SC] [expired: 2021-05-29]
      C1CF 6E31 E6BA DE88 68B1  72B4 F42E D6FB AB17 C654
uid           [ expired] Open Robotics <info@osrfoundation.org>

---

While pgp-happy-eyeballs works great to ignore unresponsive servers 👍, it would be nice to have a way to improve the server list in order to use only "up-to-date" keyservers.

Workaround for users relying on pgp-happy-eyeballs and needing up-to-date keys is that they should use keyserver.ubuntu.com or keys.openpgp.org directly (both are very stable and can be fetched over tls if desired). In order to fetch from keys.openpgp.org, the key's email address must be verified there.

[tianon]

While I could update the list, I don't think it's a worthwhile use of our time to continue trying to chase this.

Mirroring my comment from docker-library/official-images#11924 (comment) a bit:

pgp-happy-eyeballs was designed for a time when SKS was relatively healthy, which really isn't the case anymore. The "gossip network" might still be alive and active, but finding a good, up-to-date list of "which servers should we consider as OK" is a very challenging problem that I don't think is worth solving.

Both hkps://keys.openpgp.org and hkps://keyserver.ubuntu.com are great, stable replacements that work well and solve the problem to the extent any of our image builds actually require. 🙈

Edit: to this end, I've added a deprecation notice in #5

[andrewgdotcom]

Four years late, I discover this issue! 🤦

The main reason the SKS servers appear stale to you is because most of the well-maintained servers started tarpitting pgp-happy-eyeballs for making excessive requests for 0xB42F6819007F00F88E364FD4036A9C25BF357DD4 (@tianon's key). keyserver.ubuntu.com and keys.openpgp.org have WAF configs with different limits, so they didn't block.

As of this morning, pgp-happy-eyeballs is making ~3k requests per hour to every keyserver, from at least 44 distinct IP addresses that I can see in my own logs, only three of which have PTR records (in AWS). All of these are being automatically tarpitted for exceeding default rate limits.

It would have been nice if someone had discussed this with the keyserver operators at an earlier stage - the irony is that the SKS keyservers have never been more reliable. Check out https://spider.pgpkeys.eu for a "good, up to date list of servers that should be considered OK". 😉