Pre-filing checklist
Component(s)
Rust OTAP dataflow (rust/otap-dataflow/)
Is your feature request related to a problem?
This is the same request as for #3452 applied to the ArrowRecords form of payload data. An example of a pluggable representation that translates well into Arrow records is NetTraceV6 which includes a mixture of all existing OpenTelemetry signals.
Proposed Solution
NetTraceV6 is a combined diagnostics format capable of rendering logs, metrics, traces, and profiles from a single normalized encoding. It is a wire-boundary which uses a stateful encoder and decoder for efficiency. This representation translates well into Arrow record batches.
Support for pluggable Arrow representations will bring several potential benefits, similar to pluggable byte representations.
Represent "flat-OTAP" a specific flattening of OTAP according to a strongly enforced denormalized schema
Consumers who want a specific flattening of OTAP data into a single-table view could carry their flattened OTAP data as a pluggable Arrow representation (e.g. Parquet).
Support experimental OTAP Profiles signal as the first Arrow plugin
Adding OTAP-Profiles can be implemented as an Arrow PDataCodec instead of a new code path in otap-dataflow (at first).
Support experimental OTAP Mixed-signal telemetry representations
Supporting NetTraceV6 completely would mean adding a pluggable Arrow representation containing approximately 14 tables, many of which overlap with the hypothetical OTAP Profiles Arrow representation mentioned above.
Alternatives Considered
No response
Additional Context
This "OTAP-Mixed(NetTraceV6)" pipeline configuration as described may be considered the in-memory representation for a kind of "flight recorder" protocol, implemented as an Arrow representation plugin for mixed-signal diagnostics.
Pre-filing checklist
Component(s)
Rust OTAP dataflow (rust/otap-dataflow/)
Is your feature request related to a problem?
This is the same request as for #3452 applied to the ArrowRecords form of payload data. An example of a pluggable representation that translates well into Arrow records is NetTraceV6 which includes a mixture of all existing OpenTelemetry signals.
Proposed Solution
NetTraceV6 is a combined diagnostics format capable of rendering logs, metrics, traces, and profiles from a single normalized encoding. It is a wire-boundary which uses a stateful encoder and decoder for efficiency. This representation translates well into Arrow record batches.
Support for pluggable Arrow representations will bring several potential benefits, similar to pluggable byte representations.
Represent "flat-OTAP" a specific flattening of OTAP according to a strongly enforced denormalized schema
Consumers who want a specific flattening of OTAP data into a single-table view could carry their flattened OTAP data as a pluggable Arrow representation (e.g. Parquet).
Support experimental OTAP Profiles signal as the first Arrow plugin
Adding OTAP-Profiles can be implemented as an Arrow PDataCodec instead of a new code path in otap-dataflow (at first).
Support experimental OTAP Mixed-signal telemetry representations
Supporting NetTraceV6 completely would mean adding a pluggable Arrow representation containing approximately 14 tables, many of which overlap with the hypothetical OTAP Profiles Arrow representation mentioned above.
Alternatives Considered
No response
Additional Context
This "OTAP-Mixed(NetTraceV6)" pipeline configuration as described may be considered the in-memory representation for a kind of "flight recorder" protocol, implemented as an Arrow representation plugin for mixed-signal diagnostics.