Add NATS instrumentation #13999

AlixBa · 2025-06-08T19:10:56Z

First, sorry for the review of 5000 rows, that's not very nice.

This PR aims at instrumenting NATS for both library and java agent, using the messaging conventions.

I identified three blocks, publish, subscribe and request. The two first ones are classic pub-sub operations. The latter is a request-reply pattern, out of the box using NATS. It basically combines publish and subscribe over a unique and temporary subject(topic).

Few things:

I left out JetStreams from the instrumentation for now (as I don't have a particular interest and I have never used)
I didn't implement any messaging metrics. I'll take a look after this PR I think.
Not sure how to instrument the request. I used a CLIENT instrumenter and played with the Context to group the publish/subscribe under the same trace. Ideally, for my usage at work, I'd like everything to be in the same trace. I guess we could control this with an option.
I feel like the java agent advices are very verbose. If there is a better way of doing it, I'll gladly modify what I wrote.

What it looks like (only tested the library part):

Publish & Subscribe

Two different traces. subscribe follows_from publish with a reference/link

Request

Unique trace. subscribe child_of publish as the code calls for request

otelbot-java-instrumentation · 2025-06-08T19:14:49Z

🔧 The result from spotlessApply was committed to the PR branch.

AlixBa · 2025-06-09T07:11:08Z

@laurit is it you I should ask a review from?

laurit

Currently there is a build failure, and a test failure.

instrumentation/nats/nats-2.21/javaagent/build.gradle.kts

.../io/opentelemetry/javaagent/instrumentation/nats/v2_21/ConnectionRequestInstrumentation.java

...in/java/io/opentelemetry/javaagent/instrumentation/nats/v2_21/NatsInstrumentationModule.java

.../java/io/opentelemetry/javaagent/instrumentation/nats/v2_21/SubscriptionInstrumentation.java

...ts-2.21/library/src/main/java/io/opentelemetry/instrumentation/nats/v2_21/NatsTelemetry.java

...brary/src/main/java/io/opentelemetry/instrumentation/nats/v2_21/OpenTelemetryConnection.java

.../main/java/io/opentelemetry/instrumentation/nats/v2_21/internal/NatsInstrumenterFactory.java

.../java/io/opentelemetry/instrumentation/nats/v2_21/internal/NatsRequestContextCustomizer.java

...y/src/test/java/io/opentelemetry/instrumentation/nats/v2_21/NatsTelemetryDispatcherTest.java

AlixBa · 2025-06-22T06:06:51Z

...y/src/main/java/io/opentelemetry/instrumentation/nats/v2_21/OpenTelemetryMessageHandler.java

+import io.opentelemetry.instrumentation.api.internal.Timer;
+import io.opentelemetry.instrumentation.nats.v2_21.internal.NatsRequest;
+
+final class OpenTelemetryMessageHandler implements MessageHandler {


this is the only one class I didn't proxy as it's a single method interface and it should be pretty stable

AlixBa · 2025-06-22T10:44:58Z

.github/scripts/check-package-names.sh

@@ -29,6 +29,9 @@ for dir in $(find instrumentation -name "*.java" | grep library/src/main/java |
  if [[ "$dir" == "instrumentation/lettuce/lettuce-5.1/library/src/main/java/io/lettuce/core/protocol" ]]; then
    continue
  fi
+  if [[ "$dir" == "instrumentation/nats/nats-2.21/library/src/main/java/io/nats/client/impl" ]]; then


I need to put OpenTelemetryDispatcherFactory in this package because it relies on NatsConnection and NatsDispatcher which are package scoped. See this comment

AlixBa · 2025-06-22T10:49:20Z

...ts-2.21/library/src/main/java/io/opentelemetry/instrumentation/nats/v2_21/NatsTelemetry.java

+  }
+
+  /** Returns a {@link Options.Builder} with instrumented {@link DispatcherFactory}. */
+  public Options.Builder wrap(Options.Builder options) {


not really sure about this, as I have to rely on a build. I think it would be better to let users do options.dispatcherFactory(...) as OpenTelemetryDispatcherFactory is now exposed

AlixBa · 2025-06-22T10:54:29Z

hey @laurit

could you have a second look at this one when you have the time?
Took all the comments and followed (I hope) a very close implementation from other messaging instrumentations.

muzzle passes locally and on previous builds, so I guess the error is transient.

AlixBa · 2025-06-22T10:55:01Z

.fossa.yml

@@ -691,6 +691,12 @@ targets:
    - type: gradle
      path: ./
      target: ':instrumentation:mongo:mongo-async-3.3:javaagent'
+    - type: gradle
+      path: ./
+      target: ':instrumentation:nats:nats-2.21:javaagent'


since it passes for 2.17.2+, should I change 2.21 to 2.17.2 everywhere? (library included)

AlixBa · 2025-06-27T08:35:05Z

might be worth checking if this needs to be applied before merging

AlixBa requested a review from a team as a code owner June 8, 2025 19:10

AlixBa mentioned this pull request Jun 8, 2025

Reuse public createDispatcher nats-io/nats.java#1326

Closed

laurit reviewed Jun 10, 2025

View reviewed changes

AlixBa and others added 11 commits June 21, 2025 16:04

Bootstrap NATS instrumentation with Connection.publish(Message) tracing

66be356

don't propagate into headers when not possible

9757a46

cover all publish methods with tests

830bf68

Instrument NATS Subscription.nextMessage

c4f11a6

Instrument NATS library 'request' method

56e3656

Instrument NATS agent 'request' method

2bb53a2

Instrument NATS library 'dispatcher/subscribe' methods

c2ca73a

Refactor tests/names & minor fixes

fa341b2

Instrument NATS agent 'dispatcher' methods

b02272e

./gradlew spotlessApply

2f78f45

review & fixes

487dffb

AlixBa force-pushed the nats-instrumentation branch from fcb32fb to 487dffb Compare June 21, 2025 14:05

fix muzzle

5c4d6da

AlixBa commented Jun 22, 2025

View reviewed changes

add captured-headers

7d1b355

AlixBa force-pushed the nats-instrumentation branch from 40e1913 to 7d1b355 Compare June 22, 2025 09:53

fix dispatcher in library instrumentation

360a21f

AlixBa force-pushed the nats-instrumentation branch from 89cd845 to 360a21f Compare June 22, 2025 10:40

AlixBa commented Jun 22, 2025

View reviewed changes

Merge branch 'main' into nats-instrumentation

4e0741d

AlixBa requested a review from laurit July 1, 2025 11:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add NATS instrumentation #13999

Add NATS instrumentation #13999

Uh oh!

AlixBa commented Jun 8, 2025 •

edited

Loading

Uh oh!

otelbot-java-instrumentation bot commented Jun 8, 2025

Uh oh!

AlixBa commented Jun 9, 2025

Uh oh!

laurit left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlixBa Jun 22, 2025

Uh oh!

AlixBa Jun 22, 2025

Uh oh!

AlixBa Jun 22, 2025

Uh oh!

AlixBa commented Jun 22, 2025

Uh oh!

AlixBa Jun 22, 2025

Uh oh!

AlixBa commented Jun 27, 2025

Uh oh!

Uh oh!

Add NATS instrumentation #13999

Are you sure you want to change the base?

Add NATS instrumentation #13999

Uh oh!

Conversation

AlixBa commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Publish & Subscribe

Request

Uh oh!

otelbot-java-instrumentation bot commented Jun 8, 2025

Uh oh!

AlixBa commented Jun 9, 2025

Uh oh!

laurit left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlixBa Jun 22, 2025

Choose a reason for hiding this comment

Uh oh!

AlixBa Jun 22, 2025

Choose a reason for hiding this comment

Uh oh!

AlixBa Jun 22, 2025

Choose a reason for hiding this comment

Uh oh!

AlixBa commented Jun 22, 2025

Uh oh!

AlixBa Jun 22, 2025

Choose a reason for hiding this comment

Uh oh!

AlixBa commented Jun 27, 2025

Uh oh!

Uh oh!

AlixBa commented Jun 8, 2025 •

edited

Loading