feat: Add default_spans_per_span_set config and treat spss=0 as unlimited

CHANGELOG.md

@@ -23,6 +23,7 @@
 * [ENHANCEMENT] Improve shutdown time in the first 30 seconds [#5725](https://github.com/grafana/tempo/pull/5725) (@ldufr)
 * [ENHANCEMENT] Add metric for tracking added latency to write requests [#5781](https://github.com/grafana/tempo/pull/5781) (@mapno)
 * [ENHANCEMENT] Improve error message when overrides fail to parse [#5787](https://github.com/grafana/tempo/pull/5787) (@mapno)
+* [ENHANCEMENT] Add `default_spans_per_span_set` configuration option to control default spans returned per trace in search results [#5858](https://github.com/grafana/tempo/pull/5858) (@iamrajiv)
 * [ENHANCEMENT] Increase weight for heavy TraceQL queries [#5782](https://github.com/grafana/tempo/pull/5782) (@ruslan-mikhailov)
 * [ENHANCEMENT] Add entity-based limiting mode for metrics-generator as an alternative to series-based limiting. [#5788](https://github.com/grafana/tempo/pull/5788) (@Logiraptor)
 * [ENHANCEMENT] Improve observability of collection failures in the metrics generator with error categorization [#5936](https://github.com/grafana/tempo/pull/5936) (@javiermolinar)
@@ -32,6 +33,7 @@
 * [ENHANCEMENT] Add "Requests Executed" panel for querier metrics in the operational dashboard. [#5848](https://github.com/grafana/tempo/pull/5848) (@anglerfishlyy)
 * [ENHANCEMENT] Add support for application/protobuf in frontend endpoints [#5865](https://github.com/grafana/tempo/pull/5865) (@oleg-kozliuk-grafana)
 * [BUGFIX] Fix compactor to properly consider SSE-KMS information during metadata copy [#5774](https://github.com/grafana/tempo/pull/5774) (@steffsas)
+* [BUGFIX] Fix `spss=0` parameter to properly mean unlimited spans instead of being rejected, and respect `max_spans_per_span_set=0` configuration [#5858](https://github.com/grafana/tempo/pull/5858) (@iamrajiv)
 * [BUGFIX] Fix incorrect results in TraceQL compare() caused by potential hash collision of string array attributes [#5835](https://github.com/grafana/tempo/pull/5835) (@mdisibio)
 * [BUGFIX] Correctly track and reject too large traces in live stores. [#5757](https://github.com/grafana/tempo/pull/5757) (@joe-elliott)
 * [BUGFIX] Fix issues related to integer dedicated columns in vParquet5-preview2 [#5716](https://github.com/grafana/tempo/pull/5716) (@stoewer)
@@ -83,7 +85,7 @@
 * [ENHANCEMENT] Use peer attributes to determine the name of a client service virtual node in the service graph. [#5381](https://github.com/grafana/tempo/pull/5381) (@martenm)
 * [ENHANCEMENT] Put actual size for writing to backend. [#5413](https://github.com/grafana/tempo/pull/5413) (@ruslan-mikhailov)
 * [ENHANCEMENT] Upgrade Azurite and Fake-gcs-server to latest version. [#5512](https://github.com/grafana/tempo/pull/5512) (@javiermolinar)
-* [ENHANCEMENT] Make block ordering deterministic. [#5411](https://github.com/grafana/tempo/pull/5411) (@rajiv-singh)
+* [ENHANCEMENT] Make block ordering deterministic. [#5411](https://github.com/grafana/tempo/pull/5411) (@iamrajiv)
 * [ENHANCEMENT] Improve exemplar selection in `quantile_over_time()`. [#5278](https://github.com/grafana/tempo/pull/5278) (@zalegrala)
 * [ENHANCEMENT] Measure bytes received before limits and publish it as `tempo_distributor_ingress_bytes_total`. [#5601](https://github.com/grafana/tempo/pull/5601) (@mapno)
 * [ENHANCEMENT] Add total size logging functionality to track trace [#5625](https://github.com/grafana/tempo/pull/5628)(@sienna011022)

docs/sources/tempo/configuration/_index.md

@@ -745,6 +745,10 @@ query_frontend:
         # The number of shards to break ingester queries into.
         [ingester_shards: <int> | default = 3]
 
+        # The default number of spans to return per span set when not specified in the request.
+        # Set to 0 to return unlimited spans by default.
+        [default_spans_per_span_set: <int> | default = 3]
+
         # The maximum allowed value of spans per span set. 0 disables this limit.
         [max_spans_per_span_set: <int> | default = 100]
 
@@ -829,15 +833,17 @@ In a similar manner, excessive queries result size can also negatively impact qu
 
 #### Limit the spans per spanset
 
-You can set the maximum spans per spanset by setting `max_spans_per_span_set` for the query-frontend.
-The default value is 100.
+You can control spans per spanset behavior using two configuration options:
+
+- `default_spans_per_span_set`: Sets the default number of spans returned when not specified in the query (default: 3). Set to `0` to return unlimited spans by default.
+- `max_spans_per_span_set`: Sets the maximum allowed value (default: 100). Set to `0` to disable the limit entirely.
 
 In Grafana or Grafana Cloud, you can use the **Span Limit** field in the [TraceQL query editor](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/data-sources/tempo/query-editor/) in Grafana Explore.
-This field sets the maximum number of spans to return for each span set.
-The maximum value that you can set for the **Span Limit** value (or the spss query) is controlled by `max_spans_per_span_set`.
+This field sets the number of spans to return for each span set (the `spss` query parameter).
+If not specified, the value from `default_spans_per_span_set` is used.
+The maximum value that you can set for **Span Limit** (or the `spss` query parameter) is controlled by `max_spans_per_span_set`.
 To disable the maximum spans per span set limit, set `max_spans_per_span_set` to `0`.
-When set to `0`, there is no maximum and users can put any value in **Span Limit**.
-However, this can only be set by a Tempo administrator, not by the user.
+When set to `0`, there is no maximum and users can request any number of spans, including unlimited spans by setting `spss=0`.
 
 #### Cap the maximum query length
 

docs/sources/tempo/configuration/manifest.md

@@ -396,6 +396,7 @@ query_frontend:
         query_ingesters_until: 30m0s
         ingester_shards: 3
         most_recent_shards: 200
+        default_spans_per_span_set: 3
         max_spans_per_span_set: 100
     trace_by_id:
         query_shards: 50

modules/frontend/config.go

@@ -91,16 +91,17 @@ func (cfg *Config) RegisterFlagsAndApplyDefaults(string, *flag.FlagSet) {
 	cfg.ResponseConsumers = 10
 	cfg.Search = SearchConfig{
 		Sharder: SearchSharderConfig{
-			QueryBackendAfter:     15 * time.Minute,
-			QueryIngestersUntil:   30 * time.Minute,
-			DefaultLimit:          20,
-			MaxLimit:              0,
-			MaxDuration:           168 * time.Hour, // 1 week
-			ConcurrentRequests:    defaultConcurrentRequests,
-			TargetBytesPerRequest: defaultTargetBytesPerRequest,
-			MostRecentShards:      defaultMostRecentShards,
-			IngesterShards:        3,
-			MaxSpansPerSpanSet:    100,
+			QueryBackendAfter:      15 * time.Minute,
+			QueryIngestersUntil:    30 * time.Minute,
+			DefaultLimit:           20,
+			MaxLimit:               0,
+			MaxDuration:            168 * time.Hour, // 1 week
+			ConcurrentRequests:     defaultConcurrentRequests,
+			TargetBytesPerRequest:  defaultTargetBytesPerRequest,
+			MostRecentShards:       defaultMostRecentShards,
+			IngesterShards:         3,
+			DefaultSpansPerSpanSet: 3,
+			MaxSpansPerSpanSet:     100,
 		},
 		SLO: slo,
 	}

modules/frontend/search_sharder.go

@@ -34,11 +34,12 @@ type SearchSharderConfig struct {
 	MaxLimit              uint32        `yaml:"max_result_limit"`
 	MaxDuration           time.Duration `yaml:"max_duration"`
 	// QueryBackendAfter determines when to query backend storage vs ingesters only.
-	QueryBackendAfter   time.Duration `yaml:"query_backend_after,omitempty"`
-	QueryIngestersUntil time.Duration `yaml:"query_ingesters_until,omitempty"`
-	IngesterShards      int           `yaml:"ingester_shards,omitempty"`
-	MostRecentShards    int           `yaml:"most_recent_shards,omitempty"`
-	MaxSpansPerSpanSet  uint32        `yaml:"max_spans_per_span_set,omitempty"`
+	QueryBackendAfter      time.Duration `yaml:"query_backend_after,omitempty"`
+	QueryIngestersUntil    time.Duration `yaml:"query_ingesters_until,omitempty"`
+	IngesterShards         int           `yaml:"ingester_shards,omitempty"`
+	MostRecentShards       int           `yaml:"most_recent_shards,omitempty"`
+	DefaultSpansPerSpanSet uint32        `yaml:"default_spans_per_span_set,omitempty"`
+	MaxSpansPerSpanSet     uint32        `yaml:"max_spans_per_span_set,omitempty"`
 
 	// RF1After specifies the time after which RF1 logic is applied, injected by the configuration
 	// or determined at runtime based on search request parameters.
@@ -74,7 +75,9 @@ func newAsyncSearchSharder(reader tempodb.Reader, o overrides.Interface, cfg Sea
 func (s asyncSearchSharder) RoundTrip(pipelineRequest pipeline.Request) (pipeline.Responses[combiner.PipelineResponse], error) {
 	r := pipelineRequest.HTTPRequest()
 
-	searchReq, err := api.ParseSearchRequest(r)
+	// Use configured default (defaults to 3 if not set in config)
+	// If default_spans_per_span_set=0 is explicitly configured, it means unlimited (return all matching spans)
+	searchReq, err := api.ParseSearchRequestWithDefault(r, s.cfg.DefaultSpansPerSpanSet)
 	if err != nil {
 		return pipeline.NewBadRequest(err), nil
 	}
@@ -99,6 +102,9 @@ func (s asyncSearchSharder) RoundTrip(pipelineRequest pipeline.Request) (pipelin
 		return pipeline.NewBadRequest(fmt.Errorf("range specified by start and end exceeds %s. received start=%d end=%d", maxDuration, searchReq.Start, searchReq.End)), nil
 	}
 
+	// Validate SpansPerSpanSet against MaxSpansPerSpanSet
+	// If MaxSpansPerSpanSet is 0, it means unlimited spans are allowed
+	// If MaxSpansPerSpanSet is non-zero, enforce the limit
 	if s.cfg.MaxSpansPerSpanSet != 0 && searchReq.SpansPerSpanSet > s.cfg.MaxSpansPerSpanSet {
 		return pipeline.NewBadRequest(fmt.Errorf("spans per span set exceeds %d. received %d", s.cfg.MaxSpansPerSpanSet, searchReq.SpansPerSpanSet)), nil
 	}

modules/frontend/search_sharder_test.go

@@ -1321,6 +1321,100 @@ func TestSearchSharderReturnsConsistentShards(t *testing.T) {
 	}
 }
 
+// TestDefaultSpansPerSpanSet verifies that the default_spans_per_span_set configuration
+// is properly used when no spss parameter is provided in the request
+func TestDefaultSpansPerSpanSet(t *testing.T) {
+	tests := []struct {
+		name               string
+		configDefault      uint32
+		requestSpss        string // empty means no spss param
+		expectedSpss       uint32
+		expectError        bool
+		maxSpansPerSpanSet uint32
+	}{
+		{
+			name:               "use configured default when no spss param",
+			configDefault:      10,
+			requestSpss:        "",
+			expectedSpss:       10,
+			maxSpansPerSpanSet: 100,
+		},
+		{
+			name:               "use zero as configured default (unlimited)",
+			configDefault:      0,
+			requestSpss:        "",
+			expectedSpss:       0, // 0 means unlimited when explicitly configured
+			maxSpansPerSpanSet: 0,
+		},
+		{
+			name:               "override configured default with request param",
+			configDefault:      10,
+			requestSpss:        "5",
+			expectedSpss:       5,
+			maxSpansPerSpanSet: 100,
+		},
+		{
+			name:               "spss=0 in URL means unlimited when max=0",
+			configDefault:      10,
+			requestSpss:        "0",
+			expectedSpss:       0, // 0 means unlimited, not "return 0 spans"
+			maxSpansPerSpanSet: 0, // max=0 means unlimited allowed
+		},
+		{
+			name:               "respect max_spans_per_span_set=0 (unlimited)",
+			configDefault:      10,
+			requestSpss:        "1000",
+			expectedSpss:       1000,
+			maxSpansPerSpanSet: 0, // 0 means unlimited
+		},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			// Track the actual spss value that was used
+			var capturedSpss uint32
+
+			next := pipeline.AsyncRoundTripperFunc[combiner.PipelineResponse](func(r pipeline.Request) (pipeline.Responses[combiner.PipelineResponse], error) {
+				// Parse the request to capture spss
+				req, err := api.ParseSearchRequest(r.HTTPRequest())
+				if err == nil {
+					capturedSpss = req.SpansPerSpanSet
+				}
+				return nil, nil
+			})
+
+			o, err := overrides.NewOverrides(overrides.Config{}, nil, prometheus.DefaultRegisterer)
+			require.NoError(t, err)
+
+			sharder := newAsyncSearchSharder(&mockReader{}, o, SearchSharderConfig{
+				ConcurrentRequests:     defaultConcurrentRequests,
+				TargetBytesPerRequest:  defaultTargetBytesPerRequest,
+				DefaultSpansPerSpanSet: tc.configDefault,
+				MaxSpansPerSpanSet:     tc.maxSpansPerSpanSet,
+			}, log.NewNopLogger())
+			testRT := sharder.Wrap(next)
+
+			// Build request URL
+			urlPath := "/"
+			if tc.requestSpss != "" {
+				urlPath = "/?spss=" + tc.requestSpss
+			}
+
+			req := httptest.NewRequest("GET", urlPath, nil)
+			req = req.WithContext(user.InjectOrgID(req.Context(), "test-tenant"))
+
+			_, err = testRT.RoundTrip(pipeline.NewHTTPRequest(req))
+
+			if tc.expectError {
+				require.Error(t, err)
+			} else {
+				require.NoError(t, err)
+				assert.Equal(t, tc.expectedSpss, capturedSpss, "spss value mismatch")
+			}
+		})
+	}
+}
+
 func urisEqual(t *testing.T, expectedURIs, actualURIs []string) {
 	require.Equal(t, len(expectedURIs), len(actualURIs))
 

pkg/api/http.go

@@ -143,6 +143,12 @@ func ParseTraceID(r *http.Request) ([]byte, error) {
 
 // ParseSearchRequest takes an http.Request and decodes query params to create a tempopb.SearchRequest
 func ParseSearchRequest(r *http.Request) (*tempopb.SearchRequest, error) {
+	return ParseSearchRequestWithDefault(r, defaultSpansPerSpanSet)
+}
+
+// ParseSearchRequestWithDefault takes an http.Request and decodes query params to create a tempopb.SearchRequest
+// using the provided default value for SpansPerSpanSet when not specified in the request
+func ParseSearchRequestWithDefault(r *http.Request, defaultSpansPerSpanSet uint32) (*tempopb.SearchRequest, error) {
 	req := &tempopb.SearchRequest{
 		Tags:            map[string]string{},
 		SpansPerSpanSet: defaultSpansPerSpanSet,
@@ -254,8 +260,8 @@ func ParseSearchRequest(r *http.Request) (*tempopb.SearchRequest, error) {
 		if err != nil {
 			return nil, fmt.Errorf("invalid spss: %w", err)
 		}
-		if spansPerSpanSet <= 0 {
-			return nil, errors.New("invalid spss: must be a positive number")
+		if spansPerSpanSet < 0 {
+			return nil, errors.New("invalid spss: must be a non-negative number")
 		}
 		req.SpansPerSpanSet = uint32(spansPerSpanSet)
 	}
@@ -743,9 +749,8 @@ func BuildSearchRequest(req *http.Request, searchReq *tempopb.SearchRequest) (*h
 	if searchReq.MinDurationMs != 0 {
 		qb.addParam(urlParamMinDuration, strconv.FormatUint(uint64(searchReq.MinDurationMs), 10)+"ms")
 	}
-	if searchReq.SpansPerSpanSet != 0 {
-		qb.addParam(urlParamSpansPerSpanSet, strconv.FormatUint(uint64(searchReq.SpansPerSpanSet), 10))
-	}
+	// Always add spans_per_span_set parameter even if 0 (which means unlimited)
+	qb.addParam(urlParamSpansPerSpanSet, strconv.FormatUint(uint64(searchReq.SpansPerSpanSet), 10))
 
 	if len(searchReq.Query) > 0 {
 		qb.addParam(urlParamQuery, searchReq.Query)

pkg/api/http_test.go

@@ -176,12 +176,15 @@ func TestQuerierParseSearchRequest(t *testing.T) {
 		{
 			name:     "zero spss",
 			urlQuery: "spss=0",
-			err:      "invalid spss: must be a positive number",
+			expected: &tempopb.SearchRequest{
+				Tags:            map[string]string{},
+				SpansPerSpanSet: 0,
+			},
 		},
 		{
 			name:     "negative spss",
 			urlQuery: "spss=-2",
-			err:      "invalid spss: must be a positive number",
+			err:      "invalid spss: must be a non-negative number",
 		},
 		{
 			name:     "non-numeric spss",
@@ -456,7 +459,7 @@ func TestBuildSearchBlockRequest(t *testing.T) {
 				Size_:         1000,
 				FooterSize:    2000,
 			},
-			query: "?start=10&end=20&limit=50&maxDuration=40ms&minDuration=30ms&tags=foo%3Dbar&blockID=b92ec614-3fd7-4299-b6db-f657e7025a9b&pagesToSearch=10&size=1000&startPage=0&encoding=s2&indexPageSize=10&totalRecords=11&dataEncoding=v1&version=v2&footerSize=2000",
+			query: "?start=10&end=20&limit=50&maxDuration=40ms&minDuration=30ms&spss=0&tags=foo%3Dbar&blockID=b92ec614-3fd7-4299-b6db-f657e7025a9b&pagesToSearch=10&size=1000&startPage=0&encoding=s2&indexPageSize=10&totalRecords=11&dataEncoding=v1&version=v2&footerSize=2000",
 		},
 		{
 			req: &tempopb.SearchBlockRequest{
@@ -598,7 +601,7 @@ func TestBuildSearchRequest(t *testing.T) {
 				MaxDurationMs: 30,
 				Limit:         50,
 			},
-			query: "?start=10&end=20&limit=50&maxDuration=30ms&tags=foo%3Dbar",
+			query: "?start=10&end=20&limit=50&maxDuration=30ms&spss=0&tags=foo%3Dbar",
 		},
 		{
 			req: &tempopb.SearchRequest{
@@ -610,7 +613,7 @@ func TestBuildSearchRequest(t *testing.T) {
 				MinDurationMs: 30,
 				Limit:         50,
 			},
-			query: "?start=10&end=20&limit=50&minDuration=30ms&tags=foo%3Dbar",
+			query: "?start=10&end=20&limit=50&minDuration=30ms&spss=0&tags=foo%3Dbar",
 		},
 		{
 			req: &tempopb.SearchRequest{
@@ -622,7 +625,7 @@ func TestBuildSearchRequest(t *testing.T) {
 				MinDurationMs: 30,
 				MaxDurationMs: 40,
 			},
-			query: "?start=10&end=20&maxDuration=40ms&minDuration=30ms&tags=foo%3Dbar",
+			query: "?start=10&end=20&maxDuration=40ms&minDuration=30ms&spss=0&tags=foo%3Dbar",
 		},
 		{
 			req: &tempopb.SearchRequest{
@@ -632,15 +635,15 @@ func TestBuildSearchRequest(t *testing.T) {
 				MinDurationMs: 30,
 				MaxDurationMs: 40,
 			},
-			query: "?start=10&end=20&maxDuration=40ms&minDuration=30ms",
+			query: "?start=10&end=20&maxDuration=40ms&minDuration=30ms&spss=0",
 		},
 		{
 			req: &tempopb.SearchRequest{
 				Query: "{ foo = `bar` }",
 				Start: 10,
 				End:   20,
 			},
-			query: "?start=10&end=20&q=%7B+foo+%3D+%60bar%60+%7D",
+			query: "?start=10&end=20&spss=0&q=%7B+foo+%3D+%60bar%60+%7D",
 		},
 	}