Restore fails when Service last-applied-configuration has "healthCheckNodePort": null #9650
Description
What steps did you take and what happened:
- Cluster has a Service of type LoadBalancer with spec.externalTrafficPolicy: Local and a non-zero spec.healthCheckNodePort assigned by the apiserver
- The kubectl.kubernetes.io/last-applied-configuration annotation contains JSON with "healthCheckNodePort": null
- Backup the namespace, then restore without --preserve-nodeports.
What happened: Restore fails in the built-in Service restore item action with an error like:
Errors:
Velero: <none>
Cluster: <none>
Namespaces:
xxx: error preparing services/xxx/syslog-listener-tcp-load-balancer: rpc error: code = Unknown desc = .spec.healthCheckNodePort accessor error: <nil> is of the type <nil>, expected float64
error preparing services/xxx/syslog-listener-udp-load-balancer: rpc error: code = Unknown desc = .spec.healthCheckNodePort accessor error: <nil> is of the type <nil>, expected float64
error preparing services/xxx/trap-listener-udp-load-balancer: rpc error: code = Unknown desc = .spec.healthCheckNodePort accessor error: <nil> is of the type <nil>, expected float64
What did you expect to happen:
Restore should succeed. null in last-applied should be treated as “the user did not specify an explicit health check node port”, so auto-assigned ports can be cleared when appropriate, without failing.
The following information will help us better understand what's going on:
Enviroment: GKE
Error from velero describe
Errors:
Velero: <none>
Cluster: <none>
Namespaces:
xxx: error preparing services/xxx/syslog-listener-tcp-load-balancer: rpc error: code = Unknown desc = .spec.healthCheckNodePort accessor error: <nil> is of the type <nil>, expected float64
error preparing services/xxx/syslog-listener-udp-load-balancer: rpc error: code = Unknown desc = .spec.healthCheckNodePort accessor error: <nil> is of the type <nil>, expected float64
error preparing services/xxx/trap-listener-udp-load-balancer: rpc error: code = Unknown desc = .spec.healthCheckNodePort accessor error: <nil> is of the type <nil>, expected float64
Example of problem service:
apiVersion: v1
kind: Service
metadata:
annotations:
cloud.google.com/neg: '{"ingress":true}'
cloud.google.com/neg-status: '{"network_endpoint_groups":{"0":"k8s2-fi0coc1c-xxx-trap-listener-udp-load-balance-bpyk6ezf"},"zones":["us-central1-a","us-central1-b","us-central1-c"]}'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"networking.gke.io/internal-load-balancer-subnet":"subnet-gke-xxx","networking.gke.io/load-balancer-type":"Internal"},"labels":{"app.kubernetes.io/managed-by":"saasDeployer","app.kubernetes.io/name":"trap-listener","app.kubernetes.io/part-of":"resource-monitoring.fault-management.trap-listener","deployment.netcracker.com/session-id":"0416b7ae-2348-4a6a-b8c4-e404c5c892b6","name":"trap-listener-v1"},"name":"trap-listener-udp-load-balancer","namespace":"xxx"},"spec":{"externalTrafficPolicy":"Local","healthCheckNodePort":null,"internalTrafficPolicy":"Local","loadBalancerIP":"10.98.35.30","ports":[{"name":"162-udp","nodePort":31701,"port":162,"protocol":"UDP","targetPort":1162}],"selector":{"name":"trap-listener-v1"},"type":"LoadBalancer"}}
networking.gke.io/internal-load-balancer-subnet: subnet-gke-xxx
networking.gke.io/load-balancer-type: Internal
service.kubernetes.io/backend-service: k8s2-fi0coc1c-xxx-trap-listener-udp-load-balance-bpyk6ezf
service.kubernetes.io/firewall-rule: k8s2-fi0coc1c-xxx-trap-listener-udp-load-balance-bpyk6ezf
service.kubernetes.io/firewall-rule-for-hc: k8s2-fi0coc1c-xxx-trap-listener-udp-load-balance-bpyk6-fw
service.kubernetes.io/healthcheck: k8s2-fi0coc1c-xxx-trap-listener-udp-load-balance-bpyk6ezf
service.kubernetes.io/udp-forwarding-rule: k8s2-udp-fi0coc1c-xxx-trap-listener-udp-load-bala-bpyk6ezf
creationTimestamp: "2025-12-23T03:30:06Z"
finalizers:
- gke.networking.io/l4-ilb-v2
- service.kubernetes.io/load-balancer-cleanup
labels:
app.kubernetes.io/managed-by: saasDeployer
app.kubernetes.io/name: trap-listener
app.kubernetes.io/part-of: resource-monitoring.fault-management.trap-listener
deployment.netcracker.com/session-id: 0416b7ae-2348-4a6a-b8c4-e404c5c892b6
name: trap-listener-v1
name: trap-listener-udp-load-balancer
namespace: xxx
resourceVersion: "1774236815457439004"
uid: b6968d95-ee64-42ba-bf94-891f82bf6971
spec:
allocateLoadBalancerNodePorts: true
clusterIP: 10.201.2.128
clusterIPs:
- 10.201.2.128
externalTrafficPolicy: Local
healthCheckNodePort: 31743
internalTrafficPolicy: Local
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
loadBalancerIP: 10.98.35.30
ports:
- name: 162-udp
nodePort: 31701
port: 162
protocol: UDP
targetPort: 1162
selector:
name: trap-listener-v1
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 10.98.35.30
ipMode: VI
Anything else you would like to add:
Affected code: deleteHealthCheckNodePort in service_action.go, NestedFloat64 on the unmarshalled last-applied object.
Suggested direction: use NestedFieldNoCopy (or similar); if the value is missing or nil, do not treat it as an explicit port; if it is a number, keep current behavior (similar to nodePort handling in deleteNodePorts).
Environment:
- Velero version (use
velero version): v1.17.2 - Velero features (use
velero client config get features): - - Kubernetes version (use
kubectl version): Server Version: v1.33.5-gke.2469000 - Kubernetes installer & version: -
- Cloud provider or hardware configuration: GCP
- OS (e.g. from
/etc/os-release): -
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.
- 👍 for "I would like to see this bug fixed as soon as possible"
- 👎 for "There are more important bugs to focus on right now"