Duplicated metrics warnings on the logs when running Besu on OpenTelemetry mode

[jake-smith754]

Hello, friends!

I've been playing around with Besu and trying to use OpenTelemetry to collect metrics and expose them through Grafana. So far it's great, most of it works wonders.

Thing is that, recently, I noticed that I started getting some warning logs on the nodes once I set metrics to OTel, here are some I extracted from my running Besu instnace:

[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument transactions has recorded multiple values for the same attributes: {}
[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument bfttimerexecutor_qbft_queue_length_current has recorded multiple values for the same attributes: {}
[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument bfttimerexecutor_qbft_active_threads_current has recorded multiple values for the same attributes: {}
[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument bfttimerexecutor_qbft_pool_size_current has recorded multiple values for the same attributes: {}
[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument bfttimerexecutor_qbft_completed_tasks_total has recorded multiple values for the same attributes: {}
[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument bfttimerexecutor_qbft_submitted_tasks_total has recorded multiple values for the same attributes: {}
[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument bfttimerexecutor_qbft_queue_length_current has recorded multiple values for the same attributes: {}
[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument bfttimerexecutor_qbft_active_threads_current has recorded multiple values for the same attributes: {}
[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument bfttimerexecutor_qbft_pool_size_current has recorded multiple values for the same attributes: {}
[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument bfttimerexecutor_qbft_completed_tasks_total has recorded multiple values for the same attributes: {}
[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument bfttimerexecutor_qbft_submitted_tasks_total has recorded multiple values for the same attributes: {}
[REDACTED] | PeriodicMetricReader-1 | WARN | AsynchronousMetricsStorage | Instrument task has recorded multiple values for the same attributes: {taskName="GetHeadersFromPeerByHashTask"}

I tried setting Besu to DEBUG mode but found no extra info related to the issue. Also did some research, I suppose it's something related to Besu's threads all producing the same metrics at a short range of time, OTel then later tries to collect those and finds that the metrics are duplicated since multiple threads produced that same metrics, thus giving us those warnings we see just above. That's my guess but, of course, not really sure if that's it...

Other than that, I also found those logs that show right when Besu starts, not sure if related to that specific issue but might be worth taking a look at:

[REDACTED] | main | INFO | OpenTelemetrySystem | Starting OpenTelemetry metrics system
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by io.opentelemetry.internal.shaded.jctools.util.UnsafeAccess (file:/opt/besu/lib/opentelemetry-sdk-trace-1.47.0.jar)
WARNING: Please consider reporting this to the maintainers of class io.opentelemetry.internal.shared.jctools.util.UnsafeAccess
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release

Do you guys have any idea on what could be causing this? Is it something on Besu's side or something related to our OTel configs? Maybe a bug on Besu?

---

Expected behavior:
Besu should expose metrics to OTel just fine with no warnings

Actual behavior:
Besu does expose metrics to OTel but still gives me those warnings related to duplicated metrics

Frequency:
The warning logs show around every 60 seconds, which is the time OTel is set to collect Besu's metrics

---

Versions and technical setup

Besu version: 26.1.0 (also reproduced on 25.12.0 and 25.8.0)

Network: QBFT, 4 validator nodes, Docker Compose

Relevant Besu flags:

--metrics-enabled=true
--metrics-protocol=opentelemetry

OTel environment variables set on each node:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=besu-node<N>

Reproducing

To make it easier to reproduce, I put together a minimal Docker Compose setup where the issue is consistently happening: https://github.com/jake-smith754/besu-otel-issue

It's a 4-node QBFT network with OpenTelemetry already configured as metrics provider. No extra setup needed, just run the commands:

git clone https://github.com/jake-smith754/besu-otel-issue
cd besu-otel-issue
docker compose up -d
docker compose logs -f node1

The AsynchronousMetricStorage warnings should start appearing within the first minute or so.

---

Thanks a lot in advance, friends!

[macfarla]

claude analysis: this is a bug in besu, not how you're configuring it. however it's a low priority bug so we may not get to this in the near future.

Root Cause: Duplicate Callback Registrations

The OTel SDK warns when multiple callbacks record values for the same instrument+attributes in a single collection cycle. Besu does this in several places:

1. OpenTelemetryTimer — the biggest offender

In metrics/core/.../OpenTelemetryTimer.java, every time a timer is stopped, it calls meter.gaugeBuilder(metricName).buildWithCallback(...). This creates a new gauge callback registration for the same metric+attributes on every
single timer observation. Over time, callbacks pile up, and OTel complains about duplicates every collection cycle.

2. BFT Executor Metrics (bfttimerexecutor_qbft_*)

MonitoredExecutors registers 5 gauges (queue_length, active_threads, pool_size, completed_tasks, submitted_tasks) for the BFT timer executor. The OpenTelemetrySystem.createLabelledSuppliedGauge doesn't cache instances — if the
same metric name is registered more than once, it creates duplicate callbacks.

3. Transaction Pool Metrics (transactions)

Multiple transaction pool layers (SparseTransactions, ReadyTransactions, BaseFeePrioritizedTransactions, etc.) each independently call metrics.initTransactionCount(...) and similar methods in their constructors. Each call
creates a new OpenTelemetryGauge with its own buildWithCallback(), leading to duplicate registrations for the same instrument name.

The Core Problem

OpenTelemetrySystem lacks deduplication/caching of async instrument registrations. The Prometheus metrics backend doesn't have this issue because it uses a different registration model. The OTel backend creates a new callback
every time, and the OTel SDK rightfully warns about it.

What You Can Do

• This is safe to run in prod — the warnings are noisy but the metrics still work. OTel picks the last-recorded value, so the data is correct.
• The sun.misc.Unsafe warnings at startup are unrelated — that's an OTel SDK internal detail about JDK compatibility, harmless.
• You could suppress the AsynchronousMetricsStorage WARN logs in your OTel collector config if the noise bothers you.

What Needs to Be Fixed in Besu

1. OpenTelemetryTimer: Should register the callback once and update the recorded value, not call buildWithCallback() on every stop()
2. OpenTelemetrySystem: Should cache LabelledSuppliedMetric instances by metric name to avoid duplicate callback registrations
3. Transaction pool / executor metrics: Registration should be deduplicated or use a single callback that iterates over all layers