What is the issue?
The status controller task inside the policy container of linkerd-destination entered a crash loop on 2026-07-21, cycling through all linkerd-destination replicas with 7,591 lease transitions before going silent. Because the panic occurs inside a Tokio-spawned task (not the main process thread), the container process never exits, Kubernetes never restarts it, and the failure is invisible: restarts=0, ready=true. After the loop stopped, no status controller held the lease. New policy resources (Server, HTTPRoute, AuthorizationPolicy) deployed after the crash are never reconciled — proxies don't receive route policies and silently fall back to the server's accessPolicy default.
How can it be reproduced?
Not really sure how to re-pro it, but this is described in an old PR #10584. The only thing I can think of is to reproduce a kube-api failure (of getting/checking the lease) by just deleting the lease itself.
Logs, error output, etc
very pod that won the policy-controller-write leader election immediately panicked:
{"message":"Status controller leadership change","leader":"true",...}
thread 'tokio-rt-worker' panicked at /build/policy-controller/k8s/status/src/index.rs:267:25:
Claims watch must not be dropped: RecvError(())
thread 'tokio-rt-worker' panicked at /build/policy-controller/k8s/status/src/index.rs:407:25:
Claims watch must not be dropped: RecvError(())
The Kubernetes policy-controller-write Lease after the loop stopped:
holderIdentity: linkerd-destination-7fb5945dc5-djfwh
leaseDurationSeconds: 30
leaseTransitions: 7591
renewTime: 2026-07-22T01:15:51Z # stale by 17+ hours
All linkerd-destination containers: restarts=0, ready=true, state=running
output of linkerd check -o short
linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
issuer certificate will expire on 2026-09-02T13:51:30Z
see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
linkerd-webhooks-and-apisvc-tls
-------------------------------
‼ proxy-injector cert is valid for at least 60 days
certificate will expire on 2026-09-05T12:24:11Z
see https://linkerd.io/2/checks/#l5d-proxy-injector-webhook-cert-not-expiring-soon for hints
‼ sp-validator cert is valid for at least 60 days
certificate will expire on 2026-09-02T13:51:31Z
see https://linkerd.io/2/checks/#l5d-sp-validator-webhook-cert-not-expiring-soon for hints
‼ policy-validator cert is valid for at least 60 days
certificate will expire on 2026-09-02T13:51:35Z
see https://linkerd.io/2/checks/#l5d-policy-validator-webhook-cert-not-expiring-soon for hints
linkerd-version
---------------
‼ cli is up-to-date
is running version 25.10.7 but the latest edge version is 26.7.1
see https://linkerd.io/2/checks/#l5d-version-cli for hints
control-plane-version
---------------------
‼ control plane is up-to-date
is running version 26.5.1 but the latest edge version is 26.7.1
see https://linkerd.io/2/checks/#l5d-version-control for hints
‼ control plane and cli versions match
control plane running edge-26.5.1 but cli running edge-25.10.7
see https://linkerd.io/2/checks/#l5d-version-control for hints
linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-7fb5945dc5-2twd2 (edge-26.5.1)
* linkerd-destination-7fb5945dc5-5zq7v (edge-26.5.1)
* linkerd-destination-7fb5945dc5-bcltr (edge-26.5.1)
* linkerd-destination-7fb5945dc5-ctghq (edge-26.5.1)
* linkerd-destination-7fb5945dc5-hpsqn (edge-26.5.1)
* linkerd-identity-5496cdb844-cpf48 (edge-26.5.1)
* linkerd-identity-5496cdb844-jsnr4 (edge-26.5.1)
* linkerd-identity-5496cdb844-mgmn5 (edge-26.5.1)
* linkerd-identity-5496cdb844-n5vmw (edge-26.5.1)
* linkerd-identity-5496cdb844-xjw74 (edge-26.5.1)
* linkerd-proxy-injector-6544d4c89-7fjcq (edge-26.5.1)
* linkerd-proxy-injector-6544d4c89-gcbzw (edge-26.5.1)
* linkerd-proxy-injector-6544d4c89-mc5t9 (edge-26.5.1)
* linkerd-proxy-injector-6544d4c89-s7xcq (edge-26.5.1)
* linkerd-proxy-injector-6544d4c89-x9vnt (edge-26.5.1)
* metrics-api-87896646f-jtbdl (edge-26.5.1)
* tap-86445856-n7hf9 (edge-26.5.1)
* tap-injector-7d4d9b96bc-gd6kw (edge-26.5.1)
* web-667c4b47d6-qjnl8 (edge-26.5.1)
see https://linkerd.io/2/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
linkerd-destination-7fb5945dc5-2twd2 running edge-26.5.1 but cli running edge-25.10.7
see https://linkerd.io/2/checks/#l5d-cp-proxy-cli-version for hints
linkerd-multicluster
--------------------
‼ Link and CLI versions match
* env1: unable to determine version
* env2: unable to determine version
* env3: unable to determine version
see https://linkerd.io/2/checks/#l5d-multicluster-links-version for hints
‼ multicluster extension proxies are up-to-date
some proxies are not running the current version:
* controller-env2-554747555b-4mrpk (edge-26.5.1)
* controller-env2-554747555b-76v6w (edge-26.5.1)
* controller-env2-554747555b-j4xhq (edge-26.5.1)
* controller-env1-687f4c955d-b7v8l (edge-26.5.1)
* controller-env1-687f4c955d-rzrll (edge-26.5.1)
* controller-env1-687f4c955d-tq8wc (edge-26.5.1)
* controller-env3-7dbf6886f5-2t2n8 (edge-26.5.1)
* controller-env3-7dbf6886f5-l8ksc (edge-26.5.1)
* controller-env3-7dbf6886f5-zb67h (edge-26.5.1)
* linkerd-gateway-78dbddc4b8-42x4r (edge-26.5.1)
* linkerd-gateway-78dbddc4b8-54zvv (edge-26.5.1)
* linkerd-gateway-78dbddc4b8-pp9ln (edge-26.5.1)
* linkerd-local-service-mirror-7bd5d946cd-fccwq (edge-26.5.1)
* linkerd-local-service-mirror-7bd5d946cd-lqbxh (edge-26.5.1)
* linkerd-local-service-mirror-7bd5d946cd-w8rj5 (edge-26.5.1)
* controller-env2-554747555b-4mrpk (edge-26.5.1)
* controller-env2-554747555b-76v6w (edge-26.5.1)
* controller-env2-554747555b-j4xhq (edge-26.5.1)
* controller-env1-687f4c955d-b7v8l (edge-26.5.1)
* controller-env1-687f4c955d-rzrll (edge-26.5.1)
* controller-env1-687f4c955d-tq8wc (edge-26.5.1)
* controller-env3-7dbf6886f5-2t2n8 (edge-26.5.1)
* controller-env3-7dbf6886f5-l8ksc (edge-26.5.1)
* controller-env3-7dbf6886f5-zb67h (edge-26.5.1)
* linkerd-gateway-78dbddc4b8-42x4r (edge-26.5.1)
* linkerd-gateway-78dbddc4b8-54zvv (edge-26.5.1)
* linkerd-gateway-78dbddc4b8-pp9ln (edge-26.5.1)
* linkerd-local-service-mirror-7bd5d946cd-fccwq (edge-26.5.1)
* linkerd-local-service-mirror-7bd5d946cd-lqbxh (edge-26.5.1)
* linkerd-local-service-mirror-7bd5d946cd-w8rj5 (edge-26.5.1)
* controller-env2-554747555b-4mrpk (edge-26.5.1)
* controller-env2-554747555b-76v6w (edge-26.5.1)
* controller-env2-554747555b-j4xhq (edge-26.5.1)
* controller-env1-687f4c955d-b7v8l (edge-26.5.1)
* controller-env1-687f4c955d-rzrll (edge-26.5.1)
* controller-env1-687f4c955d-tq8wc (edge-26.5.1)
* controller-env3-7dbf6886f5-2t2n8 (edge-26.5.1)
* controller-env3-7dbf6886f5-l8ksc (edge-26.5.1)
* controller-env3-7dbf6886f5-zb67h (edge-26.5.1)
* linkerd-gateway-78dbddc4b8-42x4r (edge-26.5.1)
* linkerd-gateway-78dbddc4b8-54zvv (edge-26.5.1)
* linkerd-gateway-78dbddc4b8-pp9ln (edge-26.5.1)
* linkerd-local-service-mirror-7bd5d946cd-fccwq (edge-26.5.1)
* linkerd-local-service-mirror-7bd5d946cd-lqbxh (edge-26.5.1)
* linkerd-local-service-mirror-7bd5d946cd-w8rj5 (edge-26.5.1)
see https://linkerd.io/2/checks/#l5d-multicluster-proxy-cp-version for hints
‼ multicluster extension proxies and cli versions match
controller-env2-554747555b-4mrpk running edge-26.5.1 but cli running edge-25.10.7
see https://linkerd.io/2/checks/#l5d-multicluster-proxy-cli-version for hints
linkerd-viz
-----------
‼ tap API server cert is valid for at least 60 days
certificate will expire on 2026-09-02T13:51:36Z
see https://linkerd.io/2/checks/#l5d-tap-cert-not-expiring-soon for hints
‼ viz extension proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-7fb5945dc5-2twd2 (edge-26.5.1)
* linkerd-destination-7fb5945dc5-5zq7v (edge-26.5.1)
* linkerd-destination-7fb5945dc5-bcltr (edge-26.5.1)
* linkerd-destination-7fb5945dc5-ctghq (edge-26.5.1)
* linkerd-destination-7fb5945dc5-hpsqn (edge-26.5.1)
* linkerd-identity-5496cdb844-cpf48 (edge-26.5.1)
* linkerd-identity-5496cdb844-jsnr4 (edge-26.5.1)
* linkerd-identity-5496cdb844-mgmn5 (edge-26.5.1)
* linkerd-identity-5496cdb844-n5vmw (edge-26.5.1)
* linkerd-identity-5496cdb844-xjw74 (edge-26.5.1)
* linkerd-proxy-injector-6544d4c89-7fjcq (edge-26.5.1)
* linkerd-proxy-injector-6544d4c89-gcbzw (edge-26.5.1)
* linkerd-proxy-injector-6544d4c89-mc5t9 (edge-26.5.1)
* linkerd-proxy-injector-6544d4c89-s7xcq (edge-26.5.1)
* linkerd-proxy-injector-6544d4c89-x9vnt (edge-26.5.1)
* metrics-api-87896646f-jtbdl (edge-26.5.1)
* tap-86445856-n7hf9 (edge-26.5.1)
* tap-injector-7d4d9b96bc-gd6kw (edge-26.5.1)
* web-667c4b47d6-qjnl8 (edge-26.5.1)
see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
linkerd-destination-7fb5945dc5-2twd2 running edge-26.5.1 but cli running edge-25.10.7
see https://linkerd.io/2/checks/#l5d-viz-proxy-cli-version for hints
Status check results are √
Environment
- Kubernetes Version: v1.35.5-gke.1241004
- Cluster Environment: GKE
- Host OS: Container-Optimized OS
- Linkerd version: edge-26.5.1
Possible solution
Bubbling up some sort of endpoint that Kubernetes can use for a liveliness probe would allow k8s to restart the pod
Additional context
No response
Would you like to work on fixing this bug?
maybe
What is the issue?
The status controller task inside the policy container of linkerd-destination entered a crash loop on 2026-07-21, cycling through all linkerd-destination replicas with 7,591 lease transitions before going silent. Because the panic occurs inside a Tokio-spawned task (not the main process thread), the container process never exits, Kubernetes never restarts it, and the failure is invisible: restarts=0, ready=true. After the loop stopped, no status controller held the lease. New policy resources (Server, HTTPRoute, AuthorizationPolicy) deployed after the crash are never reconciled — proxies don't receive route policies and silently fall back to the server's accessPolicy default.
How can it be reproduced?
Not really sure how to re-pro it, but this is described in an old PR #10584. The only thing I can think of is to reproduce a kube-api failure (of getting/checking the lease) by just deleting the lease itself.
Logs, error output, etc
very pod that won the policy-controller-write leader election immediately panicked:
{"message":"Status controller leadership change","leader":"true",...}
thread 'tokio-rt-worker' panicked at /build/policy-controller/k8s/status/src/index.rs:267:25:
Claims watch must not be dropped: RecvError(())
thread 'tokio-rt-worker' panicked at /build/policy-controller/k8s/status/src/index.rs:407:25:
Claims watch must not be dropped: RecvError(())
The Kubernetes policy-controller-write Lease after the loop stopped:
holderIdentity: linkerd-destination-7fb5945dc5-djfwh
leaseDurationSeconds: 30
leaseTransitions: 7591
renewTime: 2026-07-22T01:15:51Z # stale by 17+ hours
All linkerd-destination containers: restarts=0, ready=true, state=running
output of
linkerd check -o shortEnvironment
Possible solution
Bubbling up some sort of endpoint that Kubernetes can use for a liveliness probe would allow k8s to restart the pod
Additional context
No response
Would you like to work on fixing this bug?
maybe