Kubespray' s Cilium upgrade fails

[prennig86]

What happened?

Cilium upgrade fails during kubespray cluster upgrade

Root cause
1: kubespray in cluster upgrade will always put on kube-proxy, that causes failure on Cilium upgrade
2: A pre installed Cilium (done by a previous version of kubespray) is not upgradable by helm due to missing helm annotations and labels

What did you expect to happen?

upgrade-cluster.yml will not fail if cilium is the default network

How can we reproduce it (as minimally and precisely as possible)?

Install by the previous version of kubespray (v2.27.0) a cluster with cilium instead of calico. Upgrade it with kubespray v2.28.0

OS

Ubuntu 22

Version of Ansible

9.13.0

Version of Python

3.10.12

Version of Kubespray (commit)

2.28.0

Network plugin used

cilium

Full inventory with variables

Command used to invoke ansible

Output of ansible run

Error: Unable to install Cilium: Unable to continue with install: ServiceAccount "cilium" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "cilium"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "kube-system"
ℹ️ Using Cilium version 1.17.3
🔮 Auto-detected kube-proxy has been installed

Anything else we need to know

I see that kubeadm is always adding kube-proxy in upgrade-cluster.yml phase even if "addon/kube-proxy" is correctly added into "kubeadm_init_phases_skip_default" variable.

[tico88612]

/triage accepted

I noticed this issue earlier as well. I'll be working on this part of the upgrade patch soon, so I'll give a temporary solution here.

As a temporary workaround, you can add the necessary annotations and labels to the existing Cilium resources so that Helm can take over management.

It will be like this:

RELEASE_NAME=cilium
RELEASE_NAMESPACE=kube-system
kubectl -n $NAMESPACE annotate $KIND $NAME meta.helm.sh/release-name=$RELEASE
kubectl -n $NAMESPACE annotate $KIND $NAME meta.helm.sh/release-namespace=$NAMESPACE
kubectl -n $NAMESPACE label $KIND $NAME app.kubernetes.io/managed-by=Helm

[tico88612]

/assign

[EduardFazliev]

Same for me, I'll test the proposed workaround.

[prennig86]

/triage accepted

I noticed this issue earlier as well. I'll be working on this part of the upgrade patch soon, so I'll give a temporary solution here.

As a temporary workaround, you can add the necessary annotations and labels to the existing Cilium resources so that Helm can take over management.

It will be like this:

RELEASE_NAME=cilium
RELEASE_NAMESPACE=kube-system
kubectl -n $NAMESPACE annotate $KIND $NAME meta.helm.sh/release-name=$RELEASE
kubectl -n $NAMESPACE annotate $KIND $NAME meta.helm.sh/release-namespace=$NAMESPACE
kubectl -n $NAMESPACE label $KIND $NAME app.kubernetes.io/managed-by=Helm

Thank you!

[vdveldet]

BE AWARE: the upgrade is breaking

I installed a fresh 2 node cluster using 2.27.0 then performed upgrade to 2.28.0, kube_network_plugin: cilium as cni, all debian 12.

Upgrade stuck at (as espected in this PR )

TASK [network_plugin/cilium : Cilium | Install] ********************************
fatal: [kubnode01]: FAILED! => {"changed": true, "cmd": ["/usr/local/bin/cilium", "install", "--version", "1.17.3", "-f", "/etc/kubernetes/cilium-values.yaml"], "delta": "0:00:01.735298", "end": "2025-05-23 09:37:17.552797", "msg": "non-zero return code", "rc": 1, "start": "2025-05-23 09:37:15.817499", "stderr": "\nError: Unable to install Cilium: Unable to continue with install: ServiceAccount \"cilium\" in namespace \"kube-system\" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key \"app.kubernetes.io/managed-by\": must be set to \"Helm\"; annotation validation error: missing key \"meta.helm.sh/release-name\": must be set to \"cilium\"; annotation validation error: missing key \"meta.helm.sh/release-namespace\": must be set to \"kube-system\"", "stderr_lines": ["", "Error: Unable to install Cilium: Unable to continue with install: ServiceAccount \"cilium\" in namespace \"kube-system\" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key \"app.kubernetes.io/managed-by\": must be set to \"Helm\"; annotation validation error: missing key \"meta.helm.sh/release-name\": must be set to \"cilium\"; annotation validation error: missing key \"meta.helm.sh/release-namespace\": must be set to \"kube-system\""], "stdout": "ℹ️  Using Cilium version 1.17.3\nℹ️  Using cluster name \"default\"\n🔮 Auto-detected kube-proxy has been installed", "stdout_lines": ["ℹ️  Using Cilium version 1.17.3", "ℹ️  Using cluster name \"default\"", "🔮 Auto-detected kube-proxy has been installed"]}

Ran the below command (some will give 'not found', but this is copy paste from out my vscode)

for sa in cilium cilium-operator;
do
  kubectl -n kube-system annotate sa $sa meta.helm.sh/release-name=cilium
  kubectl -n kube-system annotate sa $sa meta.helm.sh/release-namespace=kube-system
  kubectl -n kube-system label sa $sa app.kubernetes.io/managed-by=Helm
done

for cm in cilium-config;
do
  kubectl -n kube-system annotate ConfigMap $cm meta.helm.sh/release-name=cilium
  kubectl -n kube-system annotate ConfigMap $cm meta.helm.sh/release-namespace=kube-system
  kubectl -n kube-system label ConfigMap $cm app.kubernetes.io/managed-by=Helm
done

for culsterrole in cilium cilium-operator;
do
  kubectl annotate ClusterRole $culsterrole meta.helm.sh/release-name=cilium
  kubectl annotate ClusterRole $culsterrole meta.helm.sh/release-namespace=kube-system
  kubectl label ClusterRole $culsterrole app.kubernetes.io/managed-by=Helm
done

for clusterrollebinding in cilium cilium-operator;
do
  kubectl annotate ClusterRoleBinding $clusterrollebinding meta.helm.sh/release-name=cilium
  kubectl annotate ClusterRoleBinding $clusterrollebinding meta.helm.sh/release-namespace=kube-system
  kubectl label ClusterRoleBinding $clusterrollebinding app.kubernetes.io/managed-by=Helm
done

for daemonset in cilium cilium-operator;
do
  kubectl -n kube-system annotate DaemonSet $daemonset meta.helm.sh/release-name=cilium
  kubectl -n kube-system annotate DaemonSet $daemonset meta.helm.sh/release-namespace=kube-system
  kubectl -n kube-system label DaemonSet $daemonset app.kubernetes.io/managed-by=Helm
done

for deployment in cilium cilium-operator;
do
  kubectl -n kube-system annotate Deployment $deployment meta.helm.sh/release-name=cilium
  kubectl -n kube-system annotate Deployment $deployment meta.helm.sh/release-namespace=kube-system
  kubectl -n kube-system label Deployment $deployment app.kubernetes.io/managed-by=Helm
done

Ran the upgrade-cluster.yml again.

The task get stuck on :

TASK [upgrade/post-upgrade : Wait for cilium] **********************************
fatal: [kubnode02 -> kubnode01(192.168.0.205)]: FAILED! => {"changed": true, "cmd": ["/usr/local/bin/kubectl", "--kubeconfig", "/etc/kubernetes/admin.conf", "wait", "pod", "-n", "kube-system", "-l", "k8s-app=cilium", "--field-selector", "spec.nodeName==kubnode02", "--for=condition=Ready", "--timeout=120s"], "delta": "0:02:00.083131", "end": "2025-05-23 11:24:35.147489", "msg": "non-zero return code", "rc": 1, "start": "2025-05-23 11:22:35.064358", "stderr": "error: timed out waiting for the condition on pods/cilium-cg8kg", "stderr_lines": ["error: timed out waiting for the condition on pods/cilium-cg8kg"], "stdout": "", "stdout_lines": []}

Latest log from that pod shows:

time="2025-05-23T11:26:40.006568998Z" level=warning msg="/healthz returning unhealthy" error="1.17.3 (v1.17.3-3993bd06)    Kvstore service is not ready: Waiting for initial connection to be established" state=Warning subsys=daemon
time="2025-05-23T11:26:42.007085315Z" level=warning msg="/healthz returning unhealthy" error="1.17.3 (v1.17.3-3993bd06)    Kvstore service is not ready: Waiting for initial connection to be established" state=Warning subsys=daemon
time="2025-05-23T11:26:44.006780829Z" level=warning msg="/healthz returning unhealthy" error="1.17.3 (v1.17.3-3993bd06)    Kvstore service is not ready: Waiting for initial connection to be established" state=Warning subsys=daemon
time="2025-05-23T11:26:46.005930605Z" level=warning msg="/healthz returning unhealthy" error="1.17.3 (v1.17.3-3993bd06)    Kvstore service is not ready: Waiting for initial connection to be established" state=Warning subsys=daemon

In order to recover I ran

/usr/local/bin/cilium uninstall
cd /opt/cni && chown -R root:root bin
(
NAMESPACE=cilium-secrets
kubectl proxy &
kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize
)

Then run the cluster.yml playbook again and cluster will install.

[vdveldet]

Also be aware that after a second run of cluster.yml the execution stops at

TASK [network_plugin/cilium : Cilium | Install] ********************************
fatal: [kubnode01]: FAILED! => {"changed": true, "cmd": ["/usr/local/bin/cilium", "install", "--version", "1.17.3", "-f", "/etc/kubernetes/cilium-values.yaml"], "delta": "0:00:00.216719", "end": "2025-05-23 12:01:30.622931", "msg": "non-zero return code", "rc": 1, "start": "2025-05-23 12:01:30.406212", "stderr": "\nError: Unable to install Cilium: cannot re-use a name that is still in use", "stderr_lines": ["", "Error: Unable to install Cilium: cannot re-use a name that is still in use"], "stdout": "ℹ️  Using Cilium version 1.17.3\nℹ️  Using cluster name \"default\"\n🔮 Auto-detected kube-proxy has been installed", "stdout_lines": ["ℹ️  Using Cilium version 1.17.3", "ℹ️  Using cluster name \"default\"", "🔮 Auto-detected kube-proxy has been installed"]}

[tico88612]

A cilium upgrade is equivalent to a helm upgrade. The default behavior is merge (not replace), so there will be residual kvstore informations in Cilium. (For advanced operations, e.g., replacements, a helm is required, but I can't guarantee that a helm is available in every environment).

I apologize if the upgrade was affected by cilium's change in installation method. I'll push PR to reserve a few options to ensure the upgrade goes smoothly:

1. Skip the Cilium upgrade (no downtime, but the old method still won't get any maintenance because it's not a normal installation and not officially recommended by Cilium) (I think maybe --skip-tags cilium is ok. I'll test it.)
2. Check /opt/bin/cni permissions, remove the old resources, and reinstall (but there will be downtime, and maintenance will still be available)

Future updates will move the application installation to Helm (no more custom templates), and I'll pay more attention to the upgrade process when I send out PRs. Especially now that the number of approvers is really low (check out OWNERS_ALIASES), I really hope that this part of the update will make newcomers more willing to contribute and help upgrade the Application. (without being intimidated by a 2000+ line template.)

Any comments welcome! Thank @vdveldet provide these informations.

[prennig86]

Hello @tico88612,
in my lab I figured out another bug in my opinion:

in roles/network_plugin/cilium/templates/values.yaml.j2
metrics:
enabled: {{ cilium_hubble_metrics }}
this is wrong cause the right variable to enable hubble metrics is cilium_enable_hubble_metrics

Could you please fix this also?

Our clusters use also hubble with its metrics

BR

[elatov]

Ran into a similar issue, but I ran into some additional challenges. After cilium uninstall, I just wanted to reinstall it, but initially ran into:

> cilium uninstall
🔥 Deleting pods in cilium-test namespace...
🔥 Deleting cilium-test namespace...
⌛ Uninstalling Cilium
> cilium install --version 1.17.3 -f /etc/kubernetes/cilium-values.yaml
ℹ️  Using Cilium version 1.17.3
ℹ️  Using cluster name "default"
🔮 Auto-detected kube-proxy has been installed

Error: Unable to install Cilium: failed to create resource: roles.rbac.authorization.k8s.io "cilium-tlsinterception-secrets" is forbidden: unable to create new content in namespace cilium-secrets because it is being terminated

Describing the namespace, I saw the same issue as described in https://www.linkedin.com/pulse/why-my-kubernetes-namespace-stuck-terminating-state-how-mathijssen/, deleting the apiservice unblocked that:

> kubectl get apiservice | grep metrics
v1beta1.metrics.k8s.io            kube-system/metrics-server   True        120m

Then trying again, I saw:

> cilium install --version 1.17.3 -f /etc/kubernetes/cilium-values.yaml
ℹ️  Using Cilium version 1.17.3
ℹ️  Using cluster name "default"
🔮 Auto-detected kube-proxy has been installed

Error: Unable to install Cilium: cannot re-use a name that is still in use

I saw that the helm still had cilium installed:

> helm ls -A
NAME  	NAMESPACE  	REVISION	UPDATED                                STATUS  	CHART        	APP VERSION
cilium	kube-system	4       	2025-05-27 17:04:29.210255072 -0600 MDTdeployed	cilium-1.17.3	1.17.3

I deleted that and then the install worked:

> cilium install --version 1.17.3 -f /etc/kubernetes/cilium-values.yaml
ℹ️  Using Cilium version 1.17.3
ℹ️  Using cluster name "default"
🔮 Auto-detected kube-proxy has been installed

After that I checked out the pull request in progress:

gh pr checkout 12254

And kicked off another run to recover my cluster:

ansible-playbook -i inventory/home/hosts.yaml -b upgrade-cluster.yml -e "cilium_remove_old_resources=true" -e "kube_owner=root"

Then the upgraded worked out.

[tico88612]

@prennig86 I corrected it first, but the current naming convention requires some discussion, as it does not intuitively correspond to any of the values in the chart.