Malformed responses in OpenAI streaming instrumentation can prevent response delivery or exhaust JVM heap

[johnbley]

Describe the bug

The OpenAI streaming instrumentation uses the response-controlled choice.index() as an ArrayList position. It grows the list one element at a time until the index exists, without validating that the index is nonnegative or reasonably sized.

A negative index causes IndexOutOfBoundsException; a large positive index can cause excessive allocation and eventually OutOfMemoryError. The OpenAI SDK represents this field as an unrestricted long.

More importantly, the instrumentation processes each chunk before forwarding it to the application's stream consumer or async handler. An exception in the instrumentation therefore prevents the application from receiving a chunk that the uninstrumented SDK would otherwise deliver. For synchronous streams the exception aborts stream traversal; for asynchronous streams the application's onNext is not called and the stream completes with an error.

The instrumentation also retains all distinct streamed tool calls, and, when message-content capture is enabled, accumulates all message content and tool arguments for the lifetime of the stream.

Steps to reproduce

Configure an OpenAI-compatible mock endpoint to return this SSE response to an ordinary streaming chat-completion request:

data: {"id":"chatcmpl-test","object":"chat.completion.chunk","created":0,"model":"test","choices":[{"index":-1,"delta":{"content":"hello"},"logprobs":null,"finish_reason":null}]}
data: [DONE]

Consume the response through a wrapped client:

OpenAIClient client =
    OpenAITelemetry.create(OpenTelemetry.noop()).wrap(rawClient);

try (StreamResponse<ChatCompletionChunk> response =
    client.chat().completions().createStreaming(params)) {
  response.stream().forEach(chunk -> System.out.println("received: " + chunk));
}

The application consumer is not called; instead, the instrumentation throws IndexOutOfBoundsException.

Repeat with the unwrapped rawClient. The SDK delivers the chunk to the application.

To reproduce the heap-exhaustion case in a bounded test process, change the response index to a large positive value such as 100000000 and run with a small heap. The instrumentation attempts to grow its choice buffer to that size and fails with OutOfMemoryError before forwarding the chunk.

The same ordering exists in the async wrapper: StreamListener.onChunk runs before the application's Handler.onNext.

Expected behavior

Instrumentation should not make an otherwise consumable application response unavailable. Unexpected response fields should not cause heap-sized instrumentation allocations, and failure to collect telemetry should not prevent the application from receiving the response.

Actual behavior

A negative choice index throws from the instrumentation before the application callback is invoked. A sufficiently large index can exhaust the JVM heap. In either case the instrumented application does not receive the chunk that the baseline OpenAI SDK would deliver.

Streamed tool-call cardinality is also retained without a bound. Message content and tool arguments are retained without a bound when otel.instrumentation.genai.capture-message-content=true.

Javaagent or library instrumentation version

v2.19.0 through current main, for both the javaagent-created wrapper and direct library instrumentation.

Environment

JDK: any
OS: any

Additional context

This requires a malformed or adversarial server response, so it is low probability. It is, however, a high-impact problem (changing/degrading application behavior).

This issue was found and substantially documented by codex; I (a human) have validated that it indeed looks correct and taken some time to make it more concise.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[walleseve]

I'd like to take a stab at this, I'll aim to post a draft by end of this week.