-
Notifications
You must be signed in to change notification settings - Fork 500
fix: align envoy pod template labels with controller selector #1439
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Summary of Changes
Hello @omerap12, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request resolves a deployment issue in the Envoy Gateway by correcting a mismatch between the pod template labels and the controller's selector. The fix involves updating the gateway.yaml Helm template to ensure that the labels applied to the Envoy pods are correctly recognized and managed by the Kubernetes Deployment controller, thereby enabling successful deployment.
Highlights
- Fix Deployment Error: Addresses a critical deployment failure where the
spec.template.metadata.labelsdid not match the controller'sselector, leading to an invalid deployment configuration. - Label Alignment: Modifies the
dist/chart/templates/gateway-instance/gateway.yamlfile to ensure the pod template labels are consistent with the expected selector labels. - Helm Chart Update: Replaces the
chart.labelsHelm include withchart.selectorLabelsand explicitly addsapp.kubernetes.io/name: envoyandapp.kubernetes.io/component: proxyto the pod template labels.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
|
cc @Jeffwan , noticed this today :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request fixes a deployment failure caused by a mismatch between the Envoy pod template labels and the controller's selector. The changes align the labels to match the selector requirements. My review focuses on improving the clarity and maintainability of the label definitions in the Helm chart. I've identified some confusing overrides and a duplicated key, and suggested a refactoring to make the labels more explicit and prevent potential future issues.
af78fe7 to
0fa6b8a
Compare
|
AFAIK: are required by EnvoyProxy controller. |
| labels: | ||
| {{- include "chart.labels" . | nindent 20 }} | ||
| app.kubernetes.io/component: aibrix-gateway | ||
| {{- include "chart.selectorLabels" . | nindent 20 }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some info in: https://github.com/vllm-project/aibrix/tree/main/dist/chart#limitations
Is this a related?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not directly related. The problem is envoy managed it's labels by themselves.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought I fixed related issues here https://github.com/vllm-project/aibrix/pull/1367/files but seems not.
@omerap12 did you meet problems following the current guidance? Why our helm CI chart testing can not detect the problem?
If this is a problem on the main branch, then it's a very critical problem..
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes I followed the official instructions.
The CI has no way of knowing that something like that has happened since it's checking that chart was installed successfully and all pods are in Ready state.
Since all of that has happened it passed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@omerap12 BTW, the error logs in the issue is from which component? envoy gateway? so the aibrix envoy instance is not created, right? technically, we should be able to detect it. I will help reproduce from my side.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes it's from the envoy gateway ( sorry I didn't mention this in the description )
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes sense. I can reproduce from my side.
# latest labels
template:
metadata:
labels:
app.kubernetes.io/component: proxy
app.kubernetes.io/instance: aibrix
app.kubernetes.io/name: envoy
@omerap12 nice catch! I will merge this one and could you help send a cherry-pick PR against release-0.4? We want to cut v0.4.1 and this should be included in the patch release.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No problem, but I’m on vacation. I can do it tomorrow.
Jeffwan
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
great!
Signed-off-by: Omer Aplatony <[email protected]>
0fa6b8a to
3e14d19
Compare
Pull Request Description
This bugs fixes the following error when deployed with the docs:
2025-08-12T12:41:14.721Z ERROR infrastructure runner/runner.go:116 failed to create new infra {"runner": "infrastructure", "error": "failed to create or update deployment envoy-gateway-system/envoy-aibrix-system-aibrix-eg-903790dc: failed to create/update resource with server-side apply for obj &Deployment{ObjectMeta:{envoy-aibrix-system-aibrix-eg-903790dc envoy-gateway-system 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:aibrix-eg gateway.envoyproxy.io/owning-gateway-namespace:aibrix-system] map[] [{gateway.networking.k8s.io/v1 GatewayClass aibrix-eg 51f98123-5642-4073-ac1c-71992cdc349c <nil> <nil>}] [] []},Spec:DeploymentSpec{Replicas:*1,Selector:&v1.LabelSelector{MatchLabels:map[string]string{app.kubernetes.io/component: proxy,app.kubernetes.io/managed-by: envoy-gateway,app.kubernetes.io/name: envoy,gateway.envoyproxy.io/owning-gateway-name: aibrix-eg,gateway.envoyproxy.io/owning-gateway-namespace: aibrix-system,},MatchExpressions:[]LabelSelectorRequirement{},},Template:{{ 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:aibrix-gateway app.kubernetes.io/instance:aibrix app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:aibrix gateway.envoyproxy.io/owning-gateway-name:aibrix-eg gateway.envoyproxy.io/owning-gateway-namespace:aibrix-system] map[prometheus.io/path:/stats/prometheus prometheus.io/port:19001 prometheus.io/scrape:true] [] [] []} {[{certs {nil nil nil nil nil SecretVolumeSource{SecretName:envoy,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {sds {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:envoy-aibrix-system-aibrix-eg-903790dc,},Items:[]KeyToPath{KeyToPath{Key:xds-trusted-ca.json,Path:xds-trusted-ca.json,Mode:nil,},KeyToPath{Key:xds-certificate.json,Path:xds-certificate.json,Mode:nil,},},DefaultMode:*420,Optional:*false,} nil nil nil nil nil nil nil nil nil nil nil}}] [] [{envoy envoyproxy/envoy:v1.33.2 [envoy] [--service-cluster aibrix-system/aibrix-eg --service-node $(ENVOY_POD_NAME) --config-yaml admin:\n access_log:\n - name: envoy.access_loggers.file\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\n path: /dev/null\n address:\n socket_address:\n address: 127.0.0.1\n port_value: 19000\ncluster_manager:\n local_cluster_name: aibrix-system/aibrix-eg\nnode:\n locality:\n zone: $(ENVOY_SERVICE_ZONE)\nlayered_runtime:\n layers:\n - name: global_config\n static_layer:\n envoy.restart_features.use_eds_cache_for_ads: true\n re2.max_program_size.error_level: 4294967295\n re2.max_program_size.warn_level: 1000\ndynamic_resources:\n ads_config:\n api_type: DELTA_GRPC\n transport_api_version: V3\n grpc_services:\n - envoy_grpc:\n cluster_name: xds_cluster\n set_node_on_first_message_only: true\n lds_config:\n ads: {}\n resource_api_version: V3\n cds_config:\n ads: {}\n resource_api_version: V3\nstatic_resources:\n listeners:\n - name: envoy-gateway-proxy-stats-0.0.0.0-19001\n address:\n socket_address:\n address: '0.0.0.0'\n port_value: 19001\n protocol: TCP\n bypass_overload_manager: true\n filter_chains:\n - filters:\n - name: envoy.filters.network.http_connection_manager\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n stat_prefix: eg-stats-http\n normalize_path: true\n route_config:\n name: local_route\n virtual_hosts:\n - name: prometheus_stats\n domains:\n - \"*\"\n routes:\n - match:\n path: /stats/prometheus\n headers:\n - name: \":method\"\n string_match:\n exact: GET\n route:\n cluster: prometheus_stats\n http_filters:\n - name: envoy.filters.http.router\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n clusters:\n - name: prometheus_stats\n connect_timeout: 0.250s\n type: STATIC\n lb_policy: ROUND_ROBIN\n load_assignment:\n cluster_name: prometheus_stats\n endpoints:\n - lb_endpoints:\n - endpoint:\n address:\n socket_address:\n address: 127.0.0.1\n port_value: 19000\n - connect_timeout: 10s\n eds_cluster_config:\n eds_config:\n ads: {}\n resource_api_version: 'V3'\n service_name: aibrix-system/aibrix-eg\n load_balancing_policy:\n policies:\n - typed_extension_config:\n name: 'envoy.load_balancing_policies.least_request'\n typed_config:\n '@type': 'type.googleapis.com/envoy.extensions.load_balancing_policies.least_request.v3.LeastRequest'\n locality_lb_config:\n zone_aware_lb_config:\n min_cluster_size: '1'\n name: aibrix-system/aibrix-eg\n type: EDS\n - connect_timeout: 10s\n load_assignment:\n cluster_name: xds_cluster\n endpoints:\n - load_balancing_weight: 1\n lb_endpoints:\n - load_balancing_weight: 1\n endpoint:\n address:\n socket_address:\n address: envoy-gateway.envoy-gateway-system.svc.cluster.local.\n port_value: 18000\n typed_extension_protocol_options:\n envoy.extensions.upstreams.http.v3.HttpProtocolOptions:\n \"@type\": \"type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions\"\n explicit_http_config:\n http2_protocol_options:\n connection_keepalive:\n interval: 30s\n timeout: 5s\n name: xds_cluster\n type: STRICT_DNS\n transport_socket:\n name: envoy.transport_sockets.tls\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n common_tls_context:\n tls_params:\n tls_maximum_protocol_version: TLSv1_3\n tls_certificate_sds_secret_configs:\n - name: xds_certificate\n sds_config:\n path_config_source:\n path: /sds/xds-certificate.json\n resource_api_version: V3\n validation_context_sds_secret_config:\n name: xds_trusted_ca\n sds_config:\n path_config_source:\n path: /sds/xds-trusted-ca.json\n resource_api_version: V3\noverload_manager:\n refresh_interval: 0.25s\n resource_monitors:\n - name: \"envoy.resource_monitors.global_downstream_max_connections\"\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig\n max_active_downstream_connections: 50000\n --log-level warn --cpuset-threads --drain-strategy immediate --component-log-level misc:error --drain-time-s 60] [{metrics 0 19001 TCP } {readiness 0 19003 TCP }] [] [{ENVOY_POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_SERVICE_ZONE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.annotations['topology.kubernetes.io/zone'],},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[cpu:{{1 0} {<nil>} 1 DecimalSI} memory:{{1073741824 0} {<nil>} 1Gi BinarySI}] map[cpu:{{1 0} {<nil>} 1 DecimalSI} memory:{{1073741824 0} {<nil>} 1Gi BinarySI}] []} [] <nil> [{certs true <nil> /certs <nil> } {sds false <nil> /sds <nil> }] [] &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 19003 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 19003 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:5,SuccessThreshold:1,FailureThreshold:1,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 19003 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:30,TerminationGracePeriodSeconds:nil,} &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/shutdown/ready,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,Sleep:nil,},StopSignal:nil,} /dev/termination-log File IfNotPresent &SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[ALL],},Privileged:*false,SELinuxOptions:nil,RunAsUser:*65532,RunAsNonRoot:*true,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:*false,RunAsGroup:*65532,ProcMount:nil,WindowsOptions:nil,SeccompProfile:&SeccompProfile{Type:RuntimeDefault,LocalhostProfile:nil,},AppArmorProfile:nil,} false false false} {shutdown-manager envoyproxy/gateway:v1.2.8 [envoy-gateway] [envoy shutdown-manager] [] [] [{ENVOY_POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_SERVICE_ZONE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.annotations['topology.kubernetes.io/zone'],},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{10 -3} {<nil>} 10m DecimalSI} memory:{{33554432 0} {<nil>} BinarySI}] []} [] <nil> [] [] &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:30,TerminationGracePeriodSeconds:nil,} &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[envoy-gateway envoy shutdown],},HTTPGet:nil,TCPSocket:nil,Sleep:nil,},StopSignal:nil,} /dev/termination-log File IfNotPresent &SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[ALL],},Privileged:*false,SELinuxOptions:nil,RunAsUser:*65532,RunAsNonRoot:*true,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:*false,RunAsGroup:*65532,ProcMount:nil,WindowsOptions:nil,SeccompProfile:&SeccompProfile{Type:RuntimeDefault,LocalhostProfile:nil,},AppArmorProfile:nil,} false false false}] [] Always 0x4001c05638 <nil> ClusterFirst map[] envoy-aibrix-system-aibrix-eg-903790dc 0x4001c053fb false false false <nil> nil [] &Affinity{NodeAffinity:&NodeAffinity{RequiredDuringSchedulingIgnoredDuringExecution:nil,PreferredDuringSchedulingIgnoredDuringExecution:[]PreferredSchedulingTerm{PreferredSchedulingTerm{Weight:100,Preference:NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:nvidia.com/gpu.present,Operator:NotIn,Values:[true],},},MatchFields:[]NodeSelectorRequirement{},},},},},PodAffinity:nil,PodAntiAffinity:&PodAntiAffinity{RequiredDuringSchedulingIgnoredDuringExecution:[]PodAffinityTerm{},PreferredDuringSchedulingIgnoredDuringExecution:[]WeightedPodAffinityTerm{WeightedPodAffinityTerm{Weight:100,PodAffinityTerm:PodAffinityTerm{LabelSelector:&v1.LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{LabelSelectorRequirement{Key:app.kubernetes.io/name,Operator:In,Values:[envoy],},},},Namespaces:[],TopologyKey:kubernetes.io/hostname,NamespaceSelector:nil,MatchLabelKeys:[],MismatchLabelKeys:[],},},},},} default-scheduler [] [] <nil> nil [] <nil> <nil> <nil> map[] [] <nil> nil <nil> [] [] nil}},Strategy:DeploymentStrategy{Type:RollingUpdate,RollingUpdate:&RollingUpdateDeployment{MaxUnavailable:1,MaxSurge:1,},},MinReadySeconds:0,RevisionHistoryLimit:*10,Paused:false,ProgressDeadlineSeconds:*600,},Status:DeploymentStatus{ObservedGeneration:0,Replicas:0,UpdatedReplicas:0,AvailableReplicas:0,UnavailableReplicas:0,Conditions:[]DeploymentCondition{},ReadyReplicas:0,CollisionCount:nil,TerminatingReplicas:nil,},}: Deployment.apps \"envoy-aibrix-system-aibrix-eg-903790dc\" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{\"app.kubernetes.io/component\":\"aibrix-gateway\", \"app.kubernetes.io/instance\":\"aibrix\", \"app.kubernetes.io/managed-by\":\"envoy-gateway\", \"app.kubernetes.io/name\":\"aibrix\", \"gateway.envoyproxy.io/owning-gateway-name\":\"aibrix-eg\", \"gateway.envoyproxy.io/owning-gateway-namespace\":\"aibrix-system\"}: `selector` does not match template `labels`"}Related Issues
Resolves: #[Insert issue number(s)]
Important: Before submitting, please complete the description above and review the checklist below.
Contribution Guidelines (Expand for Details)
We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:
Pull Request Title Format
Your PR title should start with one of these prefixes to indicate the nature of the change:
[Bug]: Corrections to existing functionality[CI]: Changes to build process or CI pipeline[Docs]: Updates or additions to documentation[API]: Modifications to aibrix's API or interface[CLI]: Changes or additions to the Command Line Interface[Misc]: For changes not covered above (use sparingly)Note: For changes spanning multiple categories, use multiple prefixes in order of importance.
Submission Checklist
By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.