[r103] [TraceQL] Implement structural operators

.github/workflows/ci.yml

@@ -46,7 +46,7 @@ jobs:
       - name: Lint
         uses: golangci/golangci-lint-action@v3
         with:
-          version: v1.51.0
+          version: v1.53.3
 
   unit-tests-pkg:
     name: Test packages - pkg

.gitignore

@@ -1,15 +1,17 @@
-/tempo
-/tempo-query
-/tempo-vulture
-/bin
+.DS_Store
 .idea
 .vscode
-/dist
+/bin
 /cmd/tempo-cli/tempo-cli
-/example/docker-compose/**/tempo-data/
-/example/docker-compose/**/gcs-data/
-.DS_Store
-/tempodb/encoding/benchmark_block
 /cmd/tempo-serverless/vendor/
+/dist
+/example/docker-compose/**/gcs-data/
+/example/docker-compose/**/tempo-data/
+/operations/jsonnet/microservices/test/outputs
+/operations/jsonnet/microservices/test/vendor
 /pkg/traceql/y.output
-private-key.key
+/tempo
+/tempo-query
+/tempo-vulture
+/tempodb/encoding/benchmark_block
+private-key.key

CHANGELOG.md

@@ -2,7 +2,8 @@
 
 * [CHANGE] Make vParquet2 the default block format [#2526](https://github.com/grafana/tempo/pull/2526) (@stoewer)
 * [CHANGE] Disable tempo-query by default in Jsonnet libs. [#2462](https://github.com/grafana/tempo/pull/2462) (@electron0zero)
-* [FEATURE] New experimental API to derive on-demand RED metrics grouped by any attribute, and new metrics generator processor [#2368](https://github.com/grafana/tempo/pull/2368) [#2418](https://github.com/grafana/tempo/pull/2418) [#2424](https://github.com/grafana/tempo/pull/2424) [#2442](https://github.com/grafana/tempo/pull/2442) [#2480](https://github.com/grafana/tempo/pull/2480) [#2481](https://github.com/grafana/tempo/pull/2481) [#2501](https://github.com/grafana/tempo/pull/2501) [#2579](https://github.com/grafana/tempo/pull/2579) (@mdisibio @zalegrala)
+* [FEATURE] New experimental API to derive on-demand RED metrics grouped by any attribute, and new metrics generator processor [#2368](https://github.com/grafana/tempo/pull/2368) [#2418](https://github.com/grafana/tempo/pull/2418) [#2424](https://github.com/grafana/tempo/pull/2424) [#2442](https://github.com/grafana/tempo/pull/2442) [#2480](https://github.com/grafana/tempo/pull/2480) [#2481](https://github.com/grafana/tempo/pull/2481) [#2501](https://github.com/grafana/tempo/pull/2501) [#2579](https://github.com/grafana/tempo/pull/2579) [#2582](https://github.com/grafana/tempo/pull/2582) (@mdisibio @zalegrala)
+* [FEATURE] New TraceQL structural operators descendant (>>), child (>), and sibling (~) [#2625](https://github.com/grafana/tempo/pull/2625) (@mdisibio)
 * [ENHANCEMENT] Add capability to flush all remaining traces to backend when ingester is stopped [#2538](https://github.com/grafana/tempo/pull/2538)
 * [ENHANCEMENT] Fill parent ID column and nested set columns [#2487](https://github.com/grafana/tempo/pull/2487) (@stoewer)
 * [ENHANCEMENT] Add metrics generator config option to allow customizable ring port [#2399](https://github.com/grafana/tempo/pull/2399) (@mdisibio)
@@ -22,6 +23,39 @@
     tolerate_failed_blocks: <int>
   ```
 * [CHANGE] Upgrade memcached version in jsonnet microservices [#2466](https://github.com/grafana/tempo/pull/2466) (@zalegrala)
+* [CHANGE] **Breaking Change** Convert metrics generator from deployment to a statefulset in jsonnet [#2533](https://github.com/grafana/tempo/pull/2533) [#2467](https://github.com/grafana/tempo/pull/2647) (@zalegrala)
+To support a new `processor`, the metrics generator has been converted from a `deployment` into a `statefulset` with a PVC.  This will require manual intervention in order to migrate successfully and avoid downtime.  Note that currently both a `deployment` and a `statefulset` will be managed by the jsonnet for a period of time, after which we will delete the `deployment` from this repo and you will need to delete user-side references to the `tempo_metrics_generator_deployment`, as well as delete the `deployment` itself.
+
+First, just as with the `ingester` configuration, you will need to specify a `pvc_size` and a `pvc_storage_class` for the `metrics_generator` PVC configuration.  For example:
+```jsonnet
+{
+  _config+:: {
+    metrics_generator+: {
+      pvc_size: '10Gi',
+      pvc_storage_class: 'local-path',
+    },
+  }
+}
+```
+
+Any user-side overrides for the `tempo_metrics_generator_deployment` need to be considered for the `tempo_metrics_generator_statefulset` object.
+
+Currently, the `deplyment` replicas are set to `0` by default in the jsonnet, while the `statefulset` inherits replica configuration from the `$._config` object.  To keep the `deployment` replicas around and make the transition without an outage, you can keep the replicas by overriding the following key.
+
+```jsonnet
+  tempo_metrics_generator_deployment+:
+    { spec+: { replicas: $._config.metrics_generator.replicas } },
+```
+
+This will maintain the same number of replicas you have specified in the configuration for the `statefulset`.  Note that this will be approximately double the resource requirements for a period of time while you stabilize the ring and prepare to scale down the `deployment`.
+
+You can check memberlist either with the `tempo_memberlist_client_cluster_members_count` metric, or you can visit the `http://tempo:3200/memberlist` page to see that metrics generator instances for both the `statefulset` and `deployment` are available.
+
+Once all instances are healthy, you can begin to scale down your `deployment` and delete the above reference to the `tempo_metrics_generator_deployment`.
+
+Without handling the above, a brief outage will be incurred for the metrics-generator, but everything should be functioning again once the `statefulset` for the metrics-generator is up and available.
+
+
 * [ENHANCEMENT] Add support to filter using negated regex operator `!~` [#2410](https://github.com/grafana/tempo/pull/2410) (@kousikmitra)
 * [ENHANCEMENT] Add `prefix` configuration option to `storage.trace.azure` and `storage.trace.gcs` [#2386](https://github.com/grafana/tempo/pull/2386) (@kousikmitra)
 * [ENHANCEMENT] Add `prefix` configuration option to `storage.trace.s3` [#2362](https://github.com/grafana/tempo/pull/2362) (@kousikmitra)
@@ -62,6 +96,7 @@ To make use of filtering, configure `autocomplete_filtering_enabled`.
 * [BUGFIX] Fixes issue where matches and other spanset level attributes were not persisted to the TraceQL results. [#2490](https://github.com/grafana/tempo/pull/2490) 
 * [BUGFIX] Fixes issue where ingester search could occasionally fail with file does not exist error [#2534](https://github.com/grafana/tempo/issues/2534) (@mdisibio)
 * [BUGFIX] Tempo failed to find meta.json path after adding prefix in S3/GCS/Azure configuration. [#2585](https://github.com/grafana/tempo/issues/2585) (@WildCatFish)
+* [BUGFIX] Delay logging config warnings until the logger has been initialized [#2645](https://github.com/grafana/tempo/pull/2645) (@kvrhdn)
 * [CHANGE] **Breaking Change** Rename s3.insecure_skip_verify [#2407](https://github.com/grafana/tempo/pull/2407) (@zalegrala)
 ```yaml
 storage:
@@ -762,4 +797,4 @@ Jsonnet users will now need to specify a storage request and limit for the gener
 * [BUGFIX] S3 multi-part upload errors [#306](https://github.com/grafana/tempo/pull/325)
 * [BUGFIX] Increase Prometheus `notfound` metric on tempo-vulture. [#301](https://github.com/grafana/tempo/pull/301)
 * [BUGFIX] Return 404 if searching for a tenant id that does not exist in the backend. [#321](https://github.com/grafana/tempo/pull/321)
-* [BUGFIX] Prune in-memory blocks from missing tenants. [#314](https://github.com/grafana/tempo/pull/314)
+* [BUGFIX] Prune in-memory blocks from missing tenants. [#314](https://github.com/grafana/tempo/pull/314)

cmd/tempo-cli/cmd-query-search-tag-values.go

@@ -1,7 +1,7 @@
 package main
 
 import (
-	"github.com/grafana/tempo/pkg/util"
+	"github.com/grafana/tempo/pkg/httpclient"
 )
 
 type querySearchTagValuesCmd struct {
@@ -12,7 +12,7 @@ type querySearchTagValuesCmd struct {
 }
 
 func (cmd *querySearchTagValuesCmd) Run(_ *globalOptions) error {
-	client := util.NewClient(cmd.APIEndpoint, cmd.OrgID)
+	client := httpclient.New(cmd.APIEndpoint, cmd.OrgID)
 
 	tagValues, err := client.SearchTagValues(cmd.Tag)
 	if err != nil {

cmd/tempo-cli/cmd-query-search-tags.go

@@ -1,7 +1,7 @@
 package main
 
 import (
-	"github.com/grafana/tempo/pkg/util"
+	"github.com/grafana/tempo/pkg/httpclient"
 )
 
 type querySearchTagsCmd struct {
@@ -11,7 +11,7 @@ type querySearchTagsCmd struct {
 }
 
 func (cmd *querySearchTagsCmd) Run(_ *globalOptions) error {
-	client := util.NewClient(cmd.APIEndpoint, cmd.OrgID)
+	client := httpclient.New(cmd.APIEndpoint, cmd.OrgID)
 
 	tags, err := client.SearchTags()
 	if err != nil {

cmd/tempo-cli/cmd-query-trace-id.go

@@ -1,7 +1,7 @@
 package main
 
 import (
-	"github.com/grafana/tempo/pkg/util"
+	"github.com/grafana/tempo/pkg/httpclient"
 )
 
 type queryTraceIDCmd struct {
@@ -12,7 +12,7 @@ type queryTraceIDCmd struct {
 }
 
 func (cmd *queryTraceIDCmd) Run(_ *globalOptions) error {
-	client := util.NewClient(cmd.APIEndpoint, cmd.OrgID)
+	client := httpclient.New(cmd.APIEndpoint, cmd.OrgID)
 
 	// util.QueryTrace will only add orgID header if len(orgID) > 0
 	trace, err := client.QueryTrace(cmd.TraceID)

cmd/tempo-query/tempo/plugin.go

@@ -153,7 +153,7 @@ func tlsCipherSuites() map[string]uint16 {
 	return cipherSuites
 }
 
-func (b *Backend) GetDependencies(ctx context.Context, endTs time.Time, lookback time.Duration) ([]jaeger.DependencyLink, error) {
+func (b *Backend) GetDependencies(context.Context, time.Time, time.Duration) ([]jaeger.DependencyLink, error) {
 	return nil, nil
 }
 
@@ -237,7 +237,7 @@ func (b *Backend) GetServices(ctx context.Context) ([]string, error) {
 	return b.lookupTagValues(ctx, span, serviceSearchTag)
 }
 
-func (b *Backend) GetOperations(ctx context.Context, query jaeger_spanstore.OperationQueryParameters) ([]jaeger_spanstore.Operation, error) {
+func (b *Backend) GetOperations(ctx context.Context, _ jaeger_spanstore.OperationQueryParameters) ([]jaeger_spanstore.Operation, error) {
 	span, ctx := opentracing.StartSpanFromContext(ctx, "tempo-query.GetOperations")
 	defer span.Finish()
 
@@ -416,7 +416,7 @@ func (b *Backend) lookupTagValues(ctx context.Context, span opentracing.Span, ta
 	return searchLookupResponse.TagValues, nil
 }
 
-func (b *Backend) WriteSpan(ctx context.Context, span *jaeger.Span) error {
+func (b *Backend) WriteSpan(context.Context, *jaeger.Span) error {
 	return nil
 }
 

cmd/tempo-vulture/main.go

@@ -23,6 +23,7 @@ import (
 	"google.golang.org/grpc/credentials"
 	"google.golang.org/grpc/credentials/insecure"
 
+	"github.com/grafana/tempo/pkg/httpclient"
 	"github.com/grafana/tempo/pkg/model/trace"
 	"github.com/grafana/tempo/pkg/tempopb"
 	"github.com/grafana/tempo/pkg/util"
@@ -161,7 +162,7 @@ func main() {
 					continue
 				}
 
-				client := util.NewClient(tempoQueryURL, tempoOrgID)
+				client := httpclient.New(tempoQueryURL, tempoOrgID)
 
 				// query the trace
 				queryMetrics, err := queryTrace(client, info)
@@ -192,7 +193,7 @@ func main() {
 					continue
 				}
 
-				client := util.NewClient(tempoQueryURL, tempoOrgID)
+				client := httpclient.New(tempoQueryURL, tempoOrgID)
 
 				// query a tag we expect the trace to be found within
 				searchMetrics, err := searchTag(client, seed)
@@ -322,7 +323,7 @@ func generateRandomInt(min int64, max int64, r *rand.Rand) int64 {
 	return number
 }
 
-func searchTag(client *util.Client, seed time.Time) (traceMetrics, error) {
+func searchTag(client *httpclient.Client, seed time.Time) (traceMetrics, error) {
 	tm := traceMetrics{
 		requested: 1,
 	}
@@ -388,7 +389,7 @@ func searchTag(client *util.Client, seed time.Time) (traceMetrics, error) {
 	return tm, nil
 }
 
-func searchTraceql(client *util.Client, seed time.Time) (traceMetrics, error) {
+func searchTraceql(client *httpclient.Client, seed time.Time) (traceMetrics, error) {
 	tm := traceMetrics{
 		requested: 1,
 	}
@@ -452,7 +453,7 @@ func searchTraceql(client *util.Client, seed time.Time) (traceMetrics, error) {
 	return tm, nil
 }
 
-func queryTrace(client *util.Client, info *util.TraceInfo) (traceMetrics, error) {
+func queryTrace(client *httpclient.Client, info *util.TraceInfo) (traceMetrics, error) {
 	tm := traceMetrics{
 		requested: 1,
 	}

cmd/tempo/main.go

@@ -13,9 +13,6 @@ import (
 	"github.com/drone/envsubst"
 	"github.com/go-kit/log/level"
 	"github.com/grafana/dskit/flagext"
-	"github.com/grafana/tempo/cmd/tempo/app"
-	"github.com/grafana/tempo/cmd/tempo/build"
-	"github.com/grafana/tempo/pkg/util/log"
 	ot "github.com/opentracing/opentracing-go"
 	"github.com/pkg/errors"
 	"github.com/prometheus/client_golang/prometheus"
@@ -32,6 +29,10 @@ import (
 	tracesdk "go.opentelemetry.io/otel/sdk/trace"
 	semconv "go.opentelemetry.io/otel/semconv/v1.18.0"
 	"gopkg.in/yaml.v2"
+
+	"github.com/grafana/tempo/cmd/tempo/app"
+	"github.com/grafana/tempo/cmd/tempo/build"
+	"github.com/grafana/tempo/pkg/util/log"
 )
 
 const appName = "tempo"
@@ -55,7 +56,7 @@ func main() {
 	ballastMBs := flag.Int("mem-ballast-size-mbs", 0, "Size of memory ballast to allocate in MBs.")
 	mutexProfileFraction := flag.Int("mutex-profile-fraction", 0, "Enable mutex profiling.")
 
-	config, err := loadConfig()
+	config, warningLogs, err := loadConfig()
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "failed parsing config: %v\n", err)
 		os.Exit(1)
@@ -72,6 +73,11 @@ func main() {
 	}
 	log.InitLogger(&config.Server)
 
+	// Log warnings from loading the config now the logger is initialized
+	for _, warning := range warningLogs {
+		level.Warn(log.Logger).Log(warning...)
+	}
+
 	// Init tracer
 	var shutdownTracer func()
 	if config.UseOTelTracer {
@@ -108,23 +114,23 @@ func main() {
 	runtime.KeepAlive(ballast)
 }
 
-func configIsValid(config *app.Config) bool {
+func configIsValid(config *app.Config) (warningLogs [][]any, isValid bool) {
 	// Warn the user for suspect configurations
 	if warnings := config.CheckConfig(); len(warnings) != 0 {
-		level.Warn(log.Logger).Log("-- CONFIGURATION WARNINGS --")
+		warningLogs = append(warningLogs, []any{"msg", "-- CONFIGURATION WARNINGS --"})
 		for _, w := range warnings {
 			output := []any{"msg", w.Message}
 			if w.Explain != "" {
 				output = append(output, "explain", w.Explain)
 			}
-			level.Warn(log.Logger).Log(output...)
+			warningLogs = append(warningLogs, output)
 		}
-		return false
+		return warningLogs, false
 	}
-	return true
+	return warningLogs, true
 }
 
-func loadConfig() (*app.Config, error) {
+func loadConfig() (*app.Config, [][]any, error) {
 	const (
 		configFileOption      = "config.file"
 		configExpandEnvOption = "config.expand-env"
@@ -163,20 +169,20 @@ func loadConfig() (*app.Config, error) {
 	if configFile != "" {
 		buff, err := os.ReadFile(configFile)
 		if err != nil {
-			return nil, fmt.Errorf("failed to read configFile %s: %w", configFile, err)
+			return nil, nil, fmt.Errorf("failed to read configFile %s: %w", configFile, err)
 		}
 
 		if configExpandEnv {
 			s, err := envsubst.EvalEnv(string(buff))
 			if err != nil {
-				return nil, fmt.Errorf("failed to expand env vars from configFile %s: %w", configFile, err)
+				return nil, nil, fmt.Errorf("failed to expand env vars from configFile %s: %w", configFile, err)
 			}
 			buff = []byte(s)
 		}
 
 		err = yaml.UnmarshalStrict(buff, config)
 		if err != nil {
-			return nil, fmt.Errorf("failed to parse configFile %s: %w", configFile, err)
+			return nil, nil, fmt.Errorf("failed to parse configFile %s: %w", configFile, err)
 		}
 
 	}
@@ -200,15 +206,15 @@ func loadConfig() (*app.Config, error) {
 	}
 
 	// after finalizing the configuration, verify its validity and exit if config.verify flag is true.
-	isValid := configIsValid(config)
+	warningLogs, isValid := configIsValid(config)
 	if configVerify {
 		if !isValid {
 			os.Exit(1)
 		}
 		os.Exit(0)
 	}
 
-	return config, nil
+	return config, warningLogs, nil
 }
 
 func installOpenTracingTracer(config *app.Config) (func(), error) {

docs/sources/tempo/getting-started/_index.md

@@ -28,8 +28,7 @@ Client instrumentation is the process of adding instrumentation points in the ap
 create and offload spans.
 
 {{% admonition type="note" %}}
-To learn more about instrumentation, read [instrumentation references]({{< relref "./instrumentation" >}}) to learn how to instrument your
-favorite language for distributed tracing.
+To learn more about instrumentation, read the [Instrument for tracing]({{< relref "./instrumentation" >}}) documentation to learn how to instrument your favorite language for distributed tracing.
 {{% /admonition %}}
 
 ## Pipeline (Grafana Agent)