Kube-state-metrics future as a library

[dgrisonnet]

Kube-state-metrics has been used as a standalone library by many projects for a while now to compensate for the fact that there was no way in the past to enable metrics for CRDs. Now that custom-state-metrics is a thing, I think we should reconsider if we want to continue maintaining kube-state-metrics as a library in the future or encourage our users to move to custom-state-metrics.

Since this is a big supportability breaking change for our users, I would consider that switch to be only possible to do in a new major release.

[fpetkovski]

I think this is a great point. It is always a challenge to remove features. If we decide to proceed with removing support, maybe we can start by logging a warning that is disabled only when KSM is run from the main.go file. My main concern is that we might miss some use cases that are not covered by metrics for CRDs, and we might realize too late only when people open issues.

[dgrisonnet]

The only aspect that comes to mind that is not covered by CRD metrics today is the aggregated APIs. Since resources served by aggregated APIs are not native to Kubernetes, they are not part of the main APIs, but they are also not CRDs.
I wonder if we can even had support for that since each API has its own client which is usually generated alongside it. Maybe there is some generic list watcher / reflector that we can use, but we will need to investigate that further.

That said, if we can't add support for generating metrics for Aggregated API resources, I would rather move the library outside of the repository than keeping it in kube-state-metrics. It is confusing to have a pseudo-library and the main project in the same repository.

This is an example of a project using ksm as a library to generate aggregated API metrics: https://github.com/openshift/openshift-state-metrics/blob/master/pkg/collectors/route.go.

[mrueg]

Are there any systems that use it as a library? https://github.com/kubernetes/kube-state-metrics/network/dependents seems to be empty

[fpetkovski]

I remember we had this issue a while back: #1605

[fpetkovski]

There is also this one: #579