Skip to content

Add ability to drop JMX MBean attributes and aggregate properly #14070

Open
@SylvainJuge

Description

@SylvainJuge

With JMX Insights, we have the ability to capture JMX metrics using a simple YAML configuration file.

As found while working on #14067 , some JMX MBeans have some technical details leak into their object names, which forces to capture extra "technical metric attributes" that are not relevant for the metric consumer.

MBeans are published in a tree-like structure with optional key/value pairs, the metric values are extracted from their MBean attributes.

JMX metrics are extracted from those using one or more MBean query that can have wildcards for those key/value pairs

For example, with the Jetty application server, when we need to capture the "select count" metric:

  • org.eclipse.jetty.io:context=*,type=managedselector,id=* MBean query allows to get all the MBean instances providing the selectCount MBean attribute
  • by default 10 instances of such MBean exist, each with a distinct value of id key.
  • the current JMX implementation will attempt to capture all of those as 10 distinct metrics with the same identity unless we map the id key value as a metric attribute, hence producing 10 distinct metrics.
  • we need to provide the aggregate sum of selectCount attribute over all those instances and capture it as a single metric
  • in this particular case, in addition the context key also needs to be discarded because there is often a single transient value (thus potentially high-cardinality over time).

Here is how it looks like with VisualVM:

Image

The context key value is equal to HTTP_1_1@524d6d96, and the id key have 10 distinct values from 0 to 9.

Having the ability to discard some metrics attributes and perform relevant aggregation seems to be possible through the usage of metric views, so I think we should be able to leverage this feature and apply it for JMX metrics.

For the distinct values of id having those as metric attributes is not relevant for the metric consumer.

For the context key, we can consider this as an exception (but one we have to get by) as it goes against the recommendations for MBean names, in particular for this statement:

The name of the same object should not change between different executions of the application, so for example it should not contain items like a "JVM id" or the "MBean Server Id" that will change with each execution

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions