metrics.podAnnotations for Velero pod get applied to node agents too, can't have custom node agent annotations

[vkukk]

What steps did you take and what happened:
I have enabled annotations for Datadog agent automatic discovery for the velero pod only.

  podAnnotations:
    ad.datadoghq.com/velero.checks: |
      {
        "openmetrics": {
          "init_config": {},
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:%%port%%/metrics",
              "namespace": "velero",
              "metrics": [".*"]
            }
          ]
        }
      }

This results in node-agent pods also getting same annotations which must be bug, because for node-agent pods, there is separate option nodeAgentPodMonitor with its own annotations.

This causes Datadog agent errors in openmetrics configuration because ad.datadoghq.com/velero.checks the string velero is valid only for the main velero pod, for node-agents it should be 'node-agent' as this must match container indentifier.

Datadog agent shows openmetrics error:

=============
Autodiscovery
=============

  Enabled Features
  ================
    containerd
    cri
    docker
    kube_orchestratorexplorer
    kubernetes

  Configuration Errors
  ====================
    velero/node-agent-4mz97 (0f7ecb9a-b7b2-4439-a6bd-d1555507b2e4)
    --------------------------------------------------------------
        annotation ad.datadoghq.com/velero.checks is invalid: velero doesn't match a container identifier [node-agent]

But Velero Helm chart also supports specifying annotations for the node-agents separately. Maybe this will help? After adding following under metrics in values.yaml:

nodeAgentPodMonitor:
    enabled: true
    annotations:
      ad.datadoghq.com/node-agent.checks: |
        {
          "openmetrics": {
            "init_config": {},
            "instances": [
              {
                "openmetrics_endpoint": "http://%%host%%:%%port%%/metrics",
                "namespace": "velero_node_agent",
                "metrics": [".*"]
              }
            ]
          }
        }

After running helm upgrade velero vmware-tanzu/velero --namespace velero --values values.yaml
All custom annotations are gone now from node-agents:

$ kubectl -n velero get pod/node-agent-772xn -oyaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    checksum/secret: b9754ca4d83dab2f2e3ce5d2f763c69ed92c2298dc2737084680716557250af3
    cni.projectcalico.org/containerID: 8e10d5b6fabf2929b0409fae7ee797cd1868309b0fde2f1e329d181f1f8bb821
    cni.projectcalico.org/podIP: 10.2.2.184/32
    cni.projectcalico.org/podIPs: 10.2.2.184/32
  creationTimestamp: "2025-01-28T14:52:20Z"

Lets experiment more. What would happen if setting metrics.nodeAgentPodMonitor.enabled to false?
The main Velero pod annotations appear again on node!

$ kubectl -n velero get pod/node-agent-c7sxf -oyaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    ad.datadoghq.com/velero.checks: |
      {
        "openmetrics": {
          "init_config": {},
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:%%port%%/metrics",
              "namespace": "velero",
              "metrics": [".*"]
            }
          ]
        }
      }
    checksum/secret: b9754ca4d83dab2f2e3ce5d2f763c69ed92c2298dc2737084680716557250af3
    cni.projectcalico.org/containerID: 9703b7c563c8d803d1227d968cb97809339cb7c3c0ccee81c88b76f9913c8c3b
    cni.projectcalico.org/podIP: 10.2.2.186/32
    cni.projectcalico.org/podIPs: 10.2.2.186/32
    prometheus.io/path: /metrics
    prometheus.io/port: "8085"
    prometheus.io/scrape: "true"
  creationTimestamp: "2025-01-28T14:57:52Z"

What did you expect to happen:

1. metrics.podAnnotations should only apply to main Velero pod
2. metrics.nodeAgentPodMonitor.annotations should be applied to node agent pods.
3. setting metrics.nodeAgentPodMonitor.enabled to false should not enable node agent prometheus metrics nor write wrong annotations!

Environment:

• helm version (use helm version): version.BuildInfo{Version:"v3.15.4", GitCommit:"fa9efb07d9d8debbb4306d72af76a383895aa8c4", GitTreeState:"clean", GoVersion:"go1.22.6"}

• helm chart version and app version (use helm list -n <YOUR NAMESPACE>): helm list -n velero
  NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
  velero velero 34 2025-01-28 16:57:34.542298964 +0200 EET deployed velero-8.3.0 1.15.2

• Kubernetes version (use kubectl version): Client Version: v1.30.9
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.30.5

[vkukk]

This fixes the issue for me 03fe11e

[jenting]

#623

@vkukk is the above PR address your issue?

[jenting]

Had addressed by the #723, closing this issue.
Feel free to reopen it it the issue still presents.