Enable metrics feature for EBS CSI Driver

[duhminick]

Hello,

The EBS CSI driver vended via EKS add-ons, or Helm chart are able to emit metrics through Prometheus (relevant blog post: https://aws.amazon.com/blogs/mt/monitor-ebs-detailed-performance-statistics-with-amazon-managed-service-for-prometheus/). It appears that the CSI driver vended through OpenShift is unable to be configured to do so.

Is it possible for the generated YAML files to be updated to begin emitting the metrics (I.e. https://github.com/openshift/csi-operator/blob/main/assets/overlays/aws-ebs/generated/standalone/node.yaml)?

Example YAML files from the AWS vended driver that are not a part of the csi-operator:

1. https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/b75c575c5fbad77be4325d96e1024376f781aa33/charts/aws-ebs-csi-driver/templates/_node.tpl#L77-L79
2. https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/b75c575c5fbad77be4325d96e1024376f781aa33/charts/aws-ebs-csi-driver/templates/metrics.yaml#L43-L64

[Phaow]

Hello @duhminick , we discussed the issue in rebase openshift/aws-ebs-csi-driver#283 (comment) , upstream keep it disable as default -> https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/charts/aws-ebs-csi-driver/values.yaml#L236, so we want to keep consistent with it. BTW, is there any customer request for the feature in OSD/ROSA/ROSA HCP?
cc to @jsafrane @rhrmo

[duhminick]

Hey @Phaow, thanks for the follow up. It's on our roadmap to begin collecting these metrics for CloudWatch Agent.

We're relying on the EBS CSI Driver to do so. I understand that it's default, but customers using the AWS vended EBS CSI Driver can configure it to have the metrics. Would it not be possible to have it as a configurable option as well?

[Phaow]

@duhminick It is possible to enable it by default or optional enable. It sounds like a feature request for openshift, I'll discuss with team for opinions.

[Phaow]

@duhminick I discussed with team today. We need some investigation for the metrics collection costs(cpu/memory) from driver and the cost for tracking this in prometheus before we make the decision.
cc to @jsafrane @rhrmo

[Phaow]

/assign

[duhminick]

@Phaow Hi, was wondering if there were any updates?

[Phaow]

Hi @duhminick , after discussed with team, it should be an opt-In feature which needs openshift API changes to make this configuration as optional(not enable by default), it depends on our plan bandwidth in next release. Feel free to create a RFE then our PM will update the dev plan for it.

cc to @jsafrane @gnufied

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale