Image pull from factory.talos.dev does not work on Talos node

[aeleks]

Hi,

my Talos OS can pull docker images normally from any registry, but factory.talos.dev and I don't know how to debug the issue further. My laptop can docker pull the exact image.

Talos:
`
➜ talosctl images pull nginx -n node01.prod.gallifrey.local

➜ talosctl images pull factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1 -n node01.prod.gallifrey.local
Laptop:
➜ docker pull factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1
v1.10.1: Pulling from metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68
Digest: sha256:c2f263a4773ad32ec4a6743b7ecf898dad50625ac1c86111f78fc0d0b374dd97
Status: Image is up to date for factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1
factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1
`

When using this image with

➜ talosctl upgrade -n node01.prod.gallifrey.local --image factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1 --force

the dashboard shows

user: warning: [2025-05-09T21:36:07.039181137Z]: level=info msg=fetch failed error=failed to do request: Head "https://images.factory.talos.dev/v2/image-factory/metal-installer/ def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68/manifests/sha256:c2f263a4773ad32ec4a6743b7ecf898dad50625ac1c86111f78fc0d0b374dd97": net/http: TLS handshake timeout host=factory.talos.dev image=factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1 user: warning: [2025-05-09T21:36:07.042248137Z]: [talos] retrying error: failed to pull image "factory.talos.dev/metal-installer/ def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1": failed to resolve reference "factory.talos.dev/metal-installer/ def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1": failed to do request: Head "https://images.factory.talos.dev/v2/image-factory/metal-installer/ def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68/manifests/sha256:c2f263a4773ad32ec4a6743b7ecf898dad50625ac1c86111f78fc0d0b374dd97": net/http: TLS handshake timeout

Which does't make any sense, because my Kubernetes pods can resolve the domain, and Talos and my Laptop use the same dns servers (1 local hosted - which can actually resolve this domain, but also 8.8.8.8)

Do you have any further clues or an idea what else I can provide to you?

Thank you in advance!

[aeleks]

Background why I want that: I have an intel cpu with intel arc and i want to use hardware transcoding with the intel i915 extension

[rothgar]

Was this the first time you tried to pull that image? Images are built by the image factory and then cached in a CDN but sometimes the first pull times out.

I was able to pull the image from my computer, but if you are getting repeat errors it would be something to look into.

[aeleks]

Thank you for your quick response!

I have tried it multiple times (5-6 times) distributed over about 1 hour, since I really wanted it to work.

I also tried it right now 3 times, for about 10 minutes. Same error I am afraid.

[rothgar]

Because you're replacing the kube-proxy and unless you disabled KubePrism you should use localhost for the k8sServiceHost and port 7445 for the k8sServicePort. See the documentation at https://www.talos.dev/v1.10/kubernetes-guides/network/deploying-cilium/#without-kube-proxy

I would try without replacing kube-proxy first and see if it works.

[aeleks]

Ah I didn't notice, thank you. I will try different settings and let you know the outcome.

[aeleks]

The issue persists. I tried both:

1. Fixing k8sServiceHost and port to what the documentation says
2. re-enabling kube-proxy and disabeling kube proxy replacement on cilium

both did not work - which would surprise me given that i can pull from other repos. I will investigate the issue further in the following days.

Thank you for your help so far!

[NikoKS]

Hello @aeleks, I'm facing the same issue. Have you find any solution? Any help is appreciated

[aeleks]

@NikoKS sorry to disappoint you, but sadly i didn't figure it out.

I worked around the issue by:

• docker pull {image}
• docker tag {image} {my-own-registry}:{myimage}
• docker push {my-own-registry}:{myimage}
• and then finally use the image from my own registry to start the custom image with talosctl.

[skuffe]

Just wanna chime in and say, that were experiencing the exact same issue.. Same TLS timeouts as well. This only seem to affect factory, using ghcr.io works fine.

Our nodes are running in Azure.

Any tips on how i can possibly troubleshoot this?

[aeleks]

I don't know if it makes sense to have 2 bug reports - buuuut I guess it already exists.

Docker pull worked from me though. This is why I find it weird.

[NikoKS]

docker pull from my mac laptop also sometimes works (Need to retry like 3 times to get it to start downloading). But on my debian machine, it just never work no matter how much time I retried.

I do have a high ping number from my location to factory.talos.dev, around 300ms. So that might be the issue for me, but I'm really not sure if that's the case.

The only thing left for me is to setup a private registry like you....

[smira]

@skuffe we identified some issues we are in the process of fixing, at least right now workaround is to set MTU to 1400 on Azure.

cc @TimJones

[aeleks]

@smira setting the MTU to 1400 also helped me to get around the issues on my bare metal cluster at home too. Thank you very much!

[skuffe]

Thank you @smira i will give that a try!