Jaeger OpenTelemetry configuration

[pavolloffay]

I am creating this issue to agree on how Jaeger OpenTelemtery binaries should be configured.

The current configuration is documented in https://www.jaegertracing.io/docs/1.18/opentelemetry/. However, there are still doubts whether this is the right approach, hence we should revisit it.

I have created a doc that compares the current behavior with the single binary with no default configuration:
https://docs.google.com/document/d/1pDSsGGLSPro_PQJUITXIvaxJ26zSfkPW4g8A_0cz8sM/edit?usp=sharing

cc) @jaegertracing/jaeger-maintainers

Other configuration related issues:

• Print final configuration #2253 print final configuration
• Introduce new flags for Jaeger OTEL components #2246 introduce new flags for Jaeger OTEL components
• Should there be a default storage type in new OTelCollector based components? #2228 should there be a default storage
• Should be Jaeger flags applied to all components or all only to the default configuration #2223 should be Jaeger flags applied to all components or just to the default configuration?
• Give Jaeger configuration higher precedence over OpenTelemetry config  #2209 give Jaeger config higher precedence over OTEL config file

[pavolloffay]

cc) also @jaegertracing/helm-charts-maintainers

[objectiser]

Although the current approach has been good for migrating current users to the new component binaries/images, I do like the idea of a single custom Jaeger OTel Collector binary that is configured with a config, rather than flags.

However I don't like the idea that the component configurations have to be more fully elaborated in the operator's CR.

If we consider the Jaeger Operator to be responsible for managing a distributed tracing infrastructure based on configuring various components - where each of those components is an OpenTelemetry Collector (using the Jaeger custom build), then I think it is reasonable for the Jaeger operator to still simplify the OTelC config on behalf of the user - i.e. being able to configure storage (e.g. ES/Kafka) once and share it between the relevant component's OTelC configs.

EDIT: Obviously the "query" component would still be a separate binary/image, and the all-in-one variant a combination of query + an OTel Collector.

[naseemkullah]

I have two naive questions that can be discussed on gitter (please ping me there if that's the case) to not pollute this issue but:

1. I never quite understood the need for the two layers of agent and collector could we not just export to jaeger-otel-collector (which would then send to es/cassandra/kafka) from the instrumented service

2. Why do we need a jaeger flavored otel-collector, cant we build in the functionality to the core otel collector? import of trace data and exports to es, cassandra and kafka seem pretty general

[jpkrohling]

I left a couple of comments elsewhere (including in the doc), but here's my take: Jaeger OpenTelemetry binaries should be configured as closely as possible to the current (old) Jaeger components, at least for the 1.x and 2.x streams. The main reason is that people invested time and effort in learning Jaeger, and if we can help them preserve their investment, we should do it.

Personally, I prefer binaries that can be started without the need to be configured at all, "frustration-free packaging" if you will. OpenTelemetry (like Prometheus, I think) requires a minimal configuration file, and I always have to Google a minimal config file before using them, which might lead me to an outdated example. We can make it clear in the docs that a production-quality instance should be properly configured, even though I think this is an obvious statement.

[pavolloffay]

1. I never quite understood the need for the two layers of agent and collector could we not just export to jaeger-otel-collector (which would then send to es/cassandra/kafka) from the instrumented service

Agent and collector are indeed very similar. Agent is a lightweight component that runs on a host/node very close to the application. Then the agent forwards telemetry data to the collector which talks to the storage layer. Here are some docs https://www.jaegertracing.io/docs/1.18/architecture/

2. Why do we need a jaeger flavored otel-collector, cant we build in the functionality to the core otel collector? import of trace data and exports to es, cassandra and kafka seem pretty general

Providing a vendor-specific storage implementation in the OTEL core community does not seem like the right approach. If we move the storage impls to otel-contrib then it also makes sense to move there UI and other parts of the Jaeger project. OpenTelemetry's goal is to standardize data collection and not storage.

then I think it is reasonable for the Jaeger operator to still simplify the OTelC config on behalf of the user

Indeed, if the Jaeger OTEL binaries will accept only the full OTEL config, then the operator could do some magic and configure some components on the behalf of the user. Now this functionality is provided directly in the Jaeger OTEL binaries.

Regarding @jpkrohling's comment. I totally agree, with one addition. As the OTEL components are very flexible we should allow the same flexibility for our users - people using OpenTelemetry collector should be able to migrate to Jaeger with no hassle.

[pavolloffay]

Summary from the comments in the google doc:

• Jaeger should be easy to configure - hence the default configuration seems the right way to go
• Jaeger with default configuration can be built as a single binary with sub-commands (agent, collector, ingester) or multiple binaries (the current state)
• The operator should not over configure Jaeger binaries on behalf of the user - Jaeger is being deployed on different platforms and the configuration should be unified

If the default configuration is supported then it makes sense to support flags. Otherwise the users will have to provide the configuration file anyways. The file would be smaller though e.g.:

exporters:
  jaeger_elasticsearch:
    server_urls: http://es:9200/

vs

./jaeger-otel-collector --es.server-urls=http://es:9200/

[joe-elliott]

After looking into the configuration code some and reviewing the comments, I would propose either allowing configuration through Jaeger config or through the OpenTelemetry Collector config file but not both.

If the OTel config file is not provided then the agent, collector, or ingester just configures itself with a default otel pipeline that most closely mimics the behavior of the traditional Jaeger services and additional Jaeger config is applied on top. If the OTel config file is provided then it is loaded as is. If both are provided the process exits with error.

Configuration merging will create a long list of edge cases, undefined behaviors and issues like #2438, #2223, ...

Pros:

• Jaeger operators will be able to use the configuration they are familiar with
• OTel Collector operators will be able to drop their configuration in
• Print final configuration #2253 will allow Jaeger operators to transition to the OTel Config file if they wanted to

[yurishkuro]

The approach here is different from what we're proposing for jaeger-v2 #3500