Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ To activate the `local-blocks` processor for all users, add it to the list of pr
```yaml
# Global overrides configuration.
overrides:
metrics_generator_processors: ['local-blocks']
metrics_generator_processors: ["local-blocks"]
Comment thread
knylander-grafana marked this conversation as resolved.
Outdated
Comment thread
knylander-grafana marked this conversation as resolved.
Outdated
```

To configure the processor per tenant, use the `metrics_generator_processor` override.
Expand All @@ -49,7 +49,7 @@ Example for per-tenant in the per-tenant overrides:

```yaml
overrides:
'tenantID':
"tenantID":
Comment thread
knylander-grafana marked this conversation as resolved.
Outdated
Comment thread
knylander-grafana marked this conversation as resolved.
Outdated
metrics_generator_processors:
- local-blocks
```
Expand Down Expand Up @@ -135,3 +135,31 @@ query_frontend:
concurrent_jobs: 8
target_bytes_per_job: 1.25e+09 # ~1.25GB
```

## Sampling and performance optimization

TraceQL metrics queries support sampling hints to improve performance on large datasets.

### Sampling configuration considerations

When using sampling in your TraceQL metrics queries, consider:

- **Timeout settings:** Sampled queries run faster but may still benefit from adequate timeouts
- **Concurrent jobs:** Sampling reduces per-job processing time, allowing higher concurrency
- **Job sizing:** With sampling, smaller job sizes may be more efficient
Comment thread
knylander-grafana marked this conversation as resolved.
Outdated

Example configuration optimized for sampling:

```yaml
query_frontend:
metrics:
concurrent_jobs: 1500 # Higher concurrency with sampling
target_bytes_per_job: 1.5e+08 # Smaller jobs with sampling
```
Comment thread
knylander-grafana marked this conversation as resolved.
Outdated
Comment thread
knylander-grafana marked this conversation as resolved.
Outdated

### Sampling best practices

- Use `sample=true` for dashboard queries requiring fast refresh
- Apply fixed sampling rates for consistent approximation levels
- Avoid sampling for alerts or precise measurements
- Test sampling accuracy against your specific data patterns
Original file line number Diff line number Diff line change
Expand Up @@ -335,3 +335,54 @@ This example means the attribute `resource.cluster` had too many values.
```
{ __meta_error="__too_many_values__", resource.cluster=<nil> }
```

## Query hints and sampling
Comment thread
knylander-grafana marked this conversation as resolved.
Outdated

TraceQL metrics queries support query hints using the `with()` syntax to optimize performance and control sampling behavior.

TraceQL metrics queries support sampling hints to improve performance by processing a subset of data.

Sampling is particularly effective for:

- Aggregation queries over large datasets
- Dashboard queries requiring fast refresh
- Exploratory data analysis

{{< admonition type="note" >}}
Sampling hints only work with TraceQL metrics queries (those using functions like `rate()`, `count_over_time()`, etc.).
{{< /admonition >}}

### Adaptive sampling: `with(sample=true)`

Automatically determines optimal sampling strategy based on query selectivity and data volume.

```
{ resource.service.name="frontend" } | rate() with(sample=true)
```

- **Use case:** Heavy queries with large result sets
- **Performance:** 2-4x improvement on queries like `{ } | rate()`
- **Accuracy:** Maintains high accuracy by adapting sampling rate

#### Fixed span sampling: `with(span_sample=0.xx)`

Samples a fixed percentage of spans for span-level aggregations.

```
{ status=error } | count_over_time() with(span_sample=0.1)
```

#### Fixed trace sampling: `with(trace_sample=0.xx)`

Samples a fixed percentage of traces for trace-level aggregations.

```
{ } | count() by (resource.service.name) with(trace_sample=0.05)
```

### When to use sampling

- **Heavy aggregation queries** with large datasets
- **Exploratory analysis** where approximate results are acceptable
- **Dashboard queries** that need faster refresh times
- **Avoid sampling** for precise metrics or rare event detection
Comment thread
knylander-grafana marked this conversation as resolved.
Outdated
Loading