Skip to content
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
* [FEATURE] Add automemlimit support for automatic GOMEMLIMIT configuration. Enable with `memory.automemlimit_enabled: true`. [#6313](https://github.com/grafana/tempo/pull/6313) (@oleg-kozlyuk)
* [CHANGE] TraceQL metrics - change default step intervals to align with new vParquet5 timestamp columns [#6413](https://github.com/grafana/tempo/pull/6413) (@mdisibio)
* [CHANGE] Remove all traces of ingesters from the dashboards [#6352](https://github.com/grafana/tempo/pull/6352) (@javiermolinar)
* [FEATURE] Add new include_any filter policy for spanmetrics filter [#6392](https://github.com/grafana/tempo/pull/6392) (@javiermolinar)
* [FEATURE] Add span_multiplier_key to overrides. This allows tenants to specify the attribute key used for span multiplier values to compensate for head-based sampling. [#6260](https://github.com/grafana/tempo/pull/6260) (@carles-grafana)
* [FEATURE] **BREAKING CHANGE** Optimize TraceQL AST by rewriting conditions on the same attribute to their array equivalent [#6353](https://github.com/grafana/tempo/pull/6353) (@stoewer)
Slightly changes the array matching semantics of != and !~ operators and introduces stricter rules for regex literals.
Expand Down
2 changes: 1 addition & 1 deletion docs/sources/tempo/configuration/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2022,7 +2022,7 @@ overrides:
[intrinsic_dimensions: <map string to bool>]
[filter_policies: [
[
include/exclude:
include/include_any/exclude:
match_type: <string> # options: strict, regexp
attributes:
- key: <string>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -226,8 +226,13 @@ func (ds RatioBasedSampler) Description() string {
### Filtering

In some cases, you may want to reduce the number of metrics produced by the `spanmetrics` processor.
You can configure the processor to use an `include` filter to match criteria that must be present in the span in order to be included.
Following the include filter, you can use an `exclude` filter to reject portions of what was previously included by the filter policy.
To do so you can configure any of the following processors, in any order or combination:

- `include`: Defines a matching criteria that all spans must meet. If multiple include policies are defined, a span must match all of them to be included (logical AND).

- `include_any`: If a span matches any include_any policy, it is immediately included, bypassing the stricter `include` requirements (logical OR). This is ideal for capturing specific internal spans without opening the floodgates for all internal telemetry.

- `exclude`: If a span matches any exclude policy, it is rejected, even if it matched an inclusion rule.

Comment thread
javiermolinar marked this conversation as resolved.
Currently, only filtering by resource and span attributes with the following value types is supported.

Expand Down Expand Up @@ -308,6 +313,36 @@ metrics_generator:
In the above, we first include all spans which have a `resource.location` that begins with `eu-` with the `include` statement, and then exclude those with begin with `dev-`.
In this way, a flexible approach to filtering can be achieved to ensure that only metrics which are important are generated.

```yaml
---
metrics_generator:
processor:
span_metrics:
filter_policies:
# Only process spans from EU production environments
- include:
match_type: regex
attributes:
- key: resource.location
value: eu-.*
# Exception Rule: Allow INTERNAL spans for auth-service specifically
- include_any:
match_type: strict
attributes:
- key: kind
value: SPAN_KIND_INTERNAL
- key: resource.service.name
value: auth-service
# Drop any spans from development tiers
- exclude:
match_type: regex
attributes:
- key: resource.tier
value: dev-.*
```

In the above, we want to capture metrics for all production spans in the EU, but we also want to explicitly allow INTERNAL spans from the auth-service, which would otherwise be ignored by the include filter.

Comment thread
javiermolinar marked this conversation as resolved.
## Example

<p align="center"><img src="/media/docs/tempo/metrics/span-metrics-example.png" alt="Span metrics overview"></p>
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ metrics_generator:
[intrinsic_dimensions: <map string to bool>]
[filter_policies: [
[
include/exclude:
include/include_any/exclude:
match_type: <string> # options: strict, regexp
attributes:
- key: <string>
Expand Down
4 changes: 2 additions & 2 deletions modules/generator/validation/fields_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ func TestValidateFilterPolicies(t *testing.T) {
name: "no include or exclude",
policies: []filterconfig.FilterPolicy{{}},
expErr: true,
expErrText: "must have at least an `include` or `exclude`",
expErrText: "must have at least an `include`, `includeAny` or `exclude`",
},
{
name: "invalid match type on include",
Expand Down Expand Up @@ -240,7 +240,7 @@ func TestValidateFilterPolicies(t *testing.T) {
{},
},
expErr: true,
expErrText: "must have at least an `include` or `exclude`",
expErrText: "must have at least an `include`, `includeAny` or `exclude`",
},
}

Expand Down
15 changes: 11 additions & 4 deletions pkg/spanfilter/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@
)

type FilterPolicy struct {
Include *PolicyMatch `yaml:"include" json:"include,omitempty"`
Exclude *PolicyMatch `yaml:"exclude" json:"exclude,omitempty"`
Include *PolicyMatch `yaml:"include" json:"include,omitempty"`
IncludeAny *PolicyMatch `yaml:"include_any" json:"include_any,omitempty"`
Exclude *PolicyMatch `yaml:"exclude" json:"exclude,omitempty"`
}

type MatchType string
Expand All @@ -35,8 +36,8 @@
}

func ValidateFilterPolicy(policy FilterPolicy) error {
if policy.Include == nil && policy.Exclude == nil {
return fmt.Errorf("invalid filter policy; policies must have at least an `include` or `exclude`: %v", policy)
if policy.Include == nil && policy.IncludeAny == nil && policy.Exclude == nil {
return fmt.Errorf("invalid filter policy; policies must have at least an `include`, `includeAny` or `exclude`: %v", policy)

Check notice on line 40 in pkg/spanfilter/config/config.go

View workflow job for this annotation

GitHub Actions / Coverage Annotations

Uncovered lines

Lines 39-40 are not covered by tests
}

if policy.Include != nil {
Expand All @@ -45,6 +46,12 @@
}
}

if policy.IncludeAny != nil {
if err := ValidatePolicyMatch(policy.IncludeAny); err != nil {
return fmt.Errorf("invalid includeAny policy: %w", err)
}

Check notice on line 52 in pkg/spanfilter/config/config.go

View workflow job for this annotation

GitHub Actions / Coverage Annotations

Uncovered lines

Lines 49-52 are not covered by tests
}

if policy.Exclude != nil {
if err := ValidatePolicyMatch(policy.Exclude); err != nil {
return fmt.Errorf("invalid exclude policy: %w", err)
Expand Down
87 changes: 65 additions & 22 deletions pkg/spanfilter/spanfilter.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,18 @@
)

type SpanFilter struct {
filterPolicies []*filterPolicy
}
include []*splitPolicy
includeAny []*splitPolicy
exclude []*splitPolicy

type filterPolicy struct {
Include *splitPolicy
Exclude *splitPolicy
hasInclude bool
hasIncludeAny bool
hasExclude bool
}

// NewSpanFilter returns a SpanFilter that will filter spans based on the given filter policies.
func NewSpanFilter(filterPolicies []config.FilterPolicy) (*SpanFilter, error) {
var policies []*filterPolicy

sf := new(SpanFilter)
for _, policy := range filterPolicies {
err := config.ValidateFilterPolicy(policy)
if err != nil {
Expand All @@ -30,45 +30,88 @@
return nil, err
}

exclude, err := getSplitPolicy(policy.Exclude)
if include != nil {
sf.include = append(sf.include, include)
sf.hasInclude = true
}

includeAny, err := getSplitPolicy(policy.IncludeAny)
if err != nil {
return nil, err
}
p := filterPolicy{
Include: include,
Exclude: exclude,

if includeAny != nil {
sf.includeAny = append(sf.includeAny, includeAny)
sf.hasIncludeAny = true
}

if p.Include != nil || p.Exclude != nil {
policies = append(policies, &p)
exclude, err := getSplitPolicy(policy.Exclude)
if err != nil {
return nil, err
}

Check notice on line 51 in pkg/spanfilter/spanfilter.go

View workflow job for this annotation

GitHub Actions / Coverage Annotations

Uncovered lines

Lines 50-51 are not covered by tests
if exclude != nil {
sf.exclude = append(sf.exclude, exclude)
sf.hasExclude = true
}
}

return &SpanFilter{
filterPolicies: policies,
}, nil
return sf, nil
}

// ApplyFilterPolicy returns true if the span should be included in the metrics.
func (f *SpanFilter) ApplyFilterPolicy(rs *v1.Resource, span *tracev1.Span) bool {
// With no filter policies specified, all spans are included.
if len(f.filterPolicies) == 0 {
if !f.hasInclude && !f.hasIncludeAny && !f.hasExclude {
return true
}

for _, policy := range f.filterPolicies {
if policy.Include != nil && !policy.Include.Match(rs, span) {
return false
if f.hasExclude && f.isExcluded(rs, span) {
return false
}

if f.hasIncludeAny && f.isIncludedOnly(rs, span) {
return true
}

if f.hasInclude {
return f.isIncluded(rs, span)
}

// If we have an include_any but NO standard include, and we reached
// here, it means include_any didn't match. -> return false.
// IF NO inclusion rules exist at all -> return true.
return !f.hasIncludeAny
}

// This is different than the isIncluded. It's a VIP pass,working as an OR expression.
// if ANY policy matches the span is included
func (f *SpanFilter) isIncludedOnly(rs *v1.Resource, span *tracev1.Span) bool {
Comment thread
javiermolinar marked this conversation as resolved.
Outdated
for _, policy := range f.includeAny {
if policy.Match(rs, span) {
return true
}
}
return false
}

if policy.Exclude != nil && policy.Exclude.Match(rs, span) {
func (f *SpanFilter) isIncluded(rs *v1.Resource, span *tracev1.Span) bool {
for _, policy := range f.include {
if !policy.Match(rs, span) {
return false
}
}

return true
}

func (f *SpanFilter) isExcluded(rs *v1.Resource, span *tracev1.Span) bool {
for _, policy := range f.exclude {
if policy.Match(rs, span) {
return true
}
}
return false
}

func getSplitPolicy(policy *config.PolicyMatch) (*splitPolicy, error) {
if policy == nil {
return nil, nil
Expand Down
Loading