Expose Operator metrics out-of-the-box

[jmesnil]

The operator should leverage MicroProfile Metrics to expose metrics out-of-the-box if they are supported on the cluster.

In particular for the Quarkus extension, the smallrye-metrics extension should be installed.
If metrics are supported (by checking whether the ServiceMonitor CRD is installed), when the operator starts it should automatically create a ServiceMonitor (and a corresponding Service) that would point to its /metrics HTTP endpoints.

[metacosm]

@jmesnil could you point me to the documentation on that topic, please?

[jmesnil]

@metacosm for the (Go-based) WildFly operator, we added this feature: wildfly/wildfly-operator#25 (and the corresponding PR jmesnil/wildfly-operator@d859a33)

The whole idea is that if the application can expose Metrics (so for us, it would be having microprofile-metrics enabled), we create a ServiceMonitor resource that advertise to Prometheus how to reach the metrics endpoint.

So we check that the ServiceMonitor CRD is installed on the cluster (that can be on either OpenShift or a vanilla k8s cluster) and create all the required resources:

• a Service that exposes the metrics endpoint (:80 /metrics by default iirc)
• a ServiceMonitor that exposes this endpoint and selector labels to find the matching services

With that done, you can start using OpenShift monitoring stack to monitor Operator-controlled applications (provided you have enabled monitoring for user-defined projects)

If the operator is written in Java, we should also provide metrics for itself automatically if ServiceMonitor is available.

As the same logic applies to both the operator and the CR owned by the operator implementation, having a reusable library to do that would be great.

[csviri]

Hi @jmesnil we originally planned to use micrometer to expose metrics about the internals of the operator, see issue:
#64
It's a nice abstraction over metrics. Unfortunatelly I never get to that. I'm not very familiar with microprofile metrics, not sure how does it compare with micrometer.

[jmesnil]

@csviri I'm not sure whether using micrometer or MP metrics really matter as long as the operator sdk provides the configuration to specify the metrics endpoint that will be consumed by the ServiceMonitor.

[laxmikantbpandhare]

working on the similar issue #64