TraceQL metrics sampling for faster performance

[mdisibio]

What this PR does:
This adds a new query hint with(sample=true) that can be used with any TraceQL metrics query to return an approximate result faster. It works by applying probabilistic sampling at the storage layer, i.e. only inspecting xx% spans, or xx% traces, depending on the needs of the query.

It uses an adaptive probabilistic approach that responds to how common spans and traces matching the query are. When there is a lot of data, it lowers the sampling rate. When matches are rare it keeps the sampling rate higher, possibly never going below 100%. Therefore the performance gain depends on the query. Heavy queries such as { } | rate() are showing improvements of 2-4x. Needle in the haystack queries won't show an improvement, but also won't lose signal. The goal is for with(sample=true) to be safe to include in virtually any query, regardless of scale or selectivity.

The trick here is the new Sampler interface that has knowledge of the low-level parquet column activity, and the matched spansets being fed to the engine.

There are also 2 other hints added while we explore the viability of this feature:

• with(span_sample=0.xx) <-- Perform span-level sampling at the fixed percent. i.e. span_sample=0.01 only inspects 1% of spans.
• with(trace_sample=0.xx) <-- Same thing but for trace-level queries such as descendant or count()s

Example screenshot showing the level of accuracy I'm targeting with this: green is raw results, and yellow is sampled. The sample version was ~3x faster (21s vs 6s)

Additional Context
This is bringing back a feature #3275 that was lost when we changed the metrics job sharding to be block/rowgroup based. However, that was never very good and this new implementation has several advantages:

• It doesn't rely on trace or span IDs so it more efficient by not reading those columns, and isn't impacted by trace ID distribution.
• Span-level sampling for supported queries gives better accuracy than trace-level sampling for the same amount of work.
• It is done locally within the job, so it is transparent to the frontend.

I also explored using statistical methods to determine sample sizes targeting specific margins of error. You'll see that in the commit history, but there were too many edge cases, and the simple formula int(math.Ceil(proportion * 10)) outperformed in all cases.

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[knylander-grafana]

Marty, do we need to update the docs for this?

[joe-elliott]

very cool use of Expected/Measured to target a 10% rate of matching spans regardless of frequency.

a lot of Qs. most are for my edification :).

[joe-elliott]

how does calling Measured() after checking the start/end time impact the final results? does this mean that blocks that barely overlap the time range will be downsampled less? so the edges of the graph will have more accurate results?

should we call measured before we filter due to time range?

[mdisibio]

Good catch, this was something I tried both ways. Calling Measured() after the check is more accurate, because it's capturing the true rate of data being found. The edges of a block are where this most often occurs. It prevents the false signal of data looking more common than it really is. So this way the sampling probability stays high (starts at 100%), until there are enough spans passing the time check to make a real determination.

[mdisibio]

@knylander-grafana

Marty, do we need to update the docs for this?

Yes, we will want to document the main hints sample=true and sample=0.xx. Leaning towards the bottom of page https://grafana.com/docs/tempo/latest/traceql/metrics-queries/functions/, since they are (1) specific to metrics queries and (2) only understood after learning about the metrics functions themselves.

[mdisibio]

For reference, there is one gap that I've identified where this approach to sampling can introduce significant error: When a query involves a lot of data, and there is a large difference in the occurrence of the top and bottom series: i.e. rate() by (resource.service.name). This is present on every span, so the sampling rate will go low. Services with lots of data will remain accurate. But services that are rare, end up below a proper sampling rate and lose signal.

If you query the rare service directly { resource.service.name="rare"} | rate(), it will sample at a higher rate and return a high quality signal.

[joe-elliott]

one small Q and i think you should make debug/info unsafe hints. other than that lgtm!