Skip to content

Fix panic in query-frontend when combining results#3683

Merged
mapno merged 2 commits intografana:mainfrom
mapno:combine-panic
May 16, 2024
Merged

Fix panic in query-frontend when combining results#3683
mapno merged 2 commits intografana:mainfrom
mapno:combine-panic

Conversation

@mapno
Copy link
Copy Markdown
Contributor

@mapno mapno commented May 16, 2024

What this PR does:

Fixes a panic when combining results in the query frontend.

Panic stack trace
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1030 +0x125
2024-05-15 13:01:13.396	created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 1386
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1019 +0x8b
2024-05-15 13:01:13.396	google.golang.org/grpc.(*Server).serveStreams.func2.1()
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1784 +0xe3a
2024-05-15 13:01:13.396	google.golang.org/grpc.(*Server).handleStream(0xc000725600, {0x3161420, 0xc0006455f0}, 0xc046aef680)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1670 +0x11e7
2024-05-15 13:01:13.396	google.golang.org/grpc.(*Server).processStreamingRPC(0xc000725600, {0x3144358, 0xc046afebd0}, {0x3161420, 0xc0006455f0}, 0xc046aef680, 0xc04193e780, 0x484a9e0, 0x0)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1506 +0x85
2024-05-15 13:01:13.396	google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0}, 0xc040e11f50, 0xc046ab0c60?)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/grafana/dskit/middleware/grpc_logging.go:77 +0xab
2024-05-15 13:01:13.396	github.com/grafana/dskit/middleware.GRPCServerLog.StreamServerInterceptor({{0x31201e0?, 0xc000642b90?}, 0x60?, 0xc2?}, {0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0}, 0xc040e11f50, 0xc046af0d00)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1515 +0xb2
2024-05-15 13:01:13.396	google.golang.org/grpc.getChainStreamHandler.func1({0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0})
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/opentracing-contrib/go-grpc/server.go:114 +0x34a
2024-05-15 13:01:13.396	github.com/opentracing-contrib/go-grpc.OpenTracingStreamServerInterceptor.func1({0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0}, 0xc040e11f50, 0xc046af0d40)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1515 +0xb2
2024-05-15 13:01:13.396	google.golang.org/grpc.getChainStreamHandler.func1({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000})
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:44 +0xa4
2024-05-15 13:01:13.395	github.com/grafana/dskit/middleware.StreamServerInstrumentInterceptor.func1({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000}, 0xc040e11f50, 0xc046af0f00)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1515 +0xb2
2024-05-15 13:01:13.395	google.golang.org/grpc.getChainStreamHandler.func1({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/cmd/tempo/app/app.go:156 +0x9c
2024-05-15 13:01:13.395	github.com/grafana/tempo/cmd/tempo/app.(*App).setupAuthMiddleware.func2({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000}, 0xc040e11f50, 0x2a51190)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/grafana/dskit/middleware/grpc_auth.go:53 +0xb2
2024-05-15 13:01:13.395	github.com/grafana/dskit/middleware.StreamServerUserHeaderInterceptor({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000}, 0x7e9dc30c8ab0?, 0x2a51190)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/pkg/tempopb/tempo.pb.go:4100 +0x107
2024-05-15 13:01:13.395	github.com/grafana/tempo/pkg/tempopb._StreamingQuerier_Search_Handler({0x268c620, 0xc0416ca5b0}, {0x3153898, 0xc0474ae020})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/frontend.go:171 +0x2d
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend.(*QueryFrontend).Search(0x481c1c0?, 0x268c620?, {0x3157330?, 0xc046ab0db0?})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/search_handlers.go:60 +0x8f1
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend.newSearchStreamingGRPCHandler.func1(0xc046af0f40, {0x3157330, 0xc046ab0db0})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/pipeline/collector_grpc.go:53 +0x207
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend/pipeline.GRPCCollector[...].RoundTrip(0x20?, 0xc046aefd40)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/combiner/common.go:77 +0x357
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend/combiner.(*genericCombiner[...]).AddResponse(0x31701c0, {0x31318d8, 0xc047167230})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/combiner/search.go:31 +0xd4
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend/combiner.NewSearch.func2(0xc04194b2a0, 0xc0474ae060, {0x3139c60?, 0xc04194b2a0?})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/pkg/traceql/combine.go:24 +0x95
2024-05-15 13:01:13.395	github.com/grafana/tempo/pkg/traceql.(*MetadataCombiner).AddMetadata(0xc00049f9a0, 0xc0005ac230)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/pkg/traceql/combine.go:84 +0x1e5
2024-05-15 13:01:13.395	github.com/grafana/tempo/pkg/traceql.combineSearchResults(0xc000564460, 0xc0005ac230)
2024-05-15 13:01:13.395	goroutine 1390 [running]:
2024-05-15 13:01:13.395	
2024-05-15 13:01:13.395	panic: assignment to entry in nil map

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copy link
Copy Markdown
Member

@electron0zero electron0zero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested, it works :shipit:

@mapno mapno merged commit 805b3f5 into grafana:main May 16, 2024
@mapno mapno deleted the combine-panic branch May 16, 2024 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants