Skip to content

[Bugfix] Deadlock on invalid query to api/v2/search/tags#5607

Merged
ruslan-mikhailov merged 6 commits intografana:mainfrom
ruslan-mikhailov:bugfix/deadlock
Sep 4, 2025
Merged

[Bugfix] Deadlock on invalid query to api/v2/search/tags#5607
ruslan-mikhailov merged 6 commits intografana:mainfrom
ruslan-mikhailov:bugfix/deadlock

Conversation

@ruslan-mikhailov
Copy link
Copy Markdown
Contributor

@ruslan-mikhailov ruslan-mikhailov commented Sep 4, 2025

What this PR does

It fixes deadlock in ingesters.

  1. panic in ExecuteTagNames on nil pointer dereference, when rootExpr is nil, which can happen when query is invalid
  2. after recovering from panic, mutex unlock is not defered, which results in deadlock.

Impact: deadlock for read and write until ingester's restart for tenant who triggered panicked request to api/v2/search/tags

How to reproduce

  1. send request to api/v2/search/tags with invalid query. Example:
--> GET http://localhost:3100/api/v2/search/tags
--> query params: {'q': '{ a="test" }'}
<-- status: 500
<-- body: error querying ingesters in Querier.SearchTags: failed to execute f() for x.x.x.x:x: rpc error: code = Unknown desc = recovered in SearchTagsV2
  1. Try to search by trace id, it will always return 500. Example:
--> GET http://localhost:3100/api/traces/19a8654f9aaa456b7e45d19ea770ac03
<-- status: 500
<-- body: error querying ingesters in Querier.FindTraceByID: context deadline exceeded

Which issue(s) this PR fixes

Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@ruslan-mikhailov ruslan-mikhailov changed the title [Bugfix] [Bugfix] Deadlock on invalid query to api/v2/search/tags Sep 4, 2025
Comment thread modules/ingester/instance_search.go Outdated
Comment thread pkg/traceql/engine_test.go
@ruslan-mikhailov ruslan-mikhailov merged commit a6cd425 into grafana:main Sep 4, 2025
37 of 38 checks passed
@ruslan-mikhailov ruslan-mikhailov deleted the bugfix/deadlock branch September 4, 2025 11:46
ruslan-mikhailov added a commit to ruslan-mikhailov/tempo that referenced this pull request Sep 5, 2025
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2
ruslan-mikhailov added a commit that referenced this pull request Sep 5, 2025
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2
ie-pham pushed a commit to ie-pham/tempo that referenced this pull request Jan 15, 2026
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2
@ie-pham ie-pham added type/bug Something isn't working backport release-v2.8 labels Jan 15, 2026
ie-pham pushed a commit to ie-pham/tempo that referenced this pull request Jan 15, 2026
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2
ie-pham added a commit that referenced this pull request Jan 15, 2026
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2

Co-authored-by: Ruslan Mikhailov <195758209+ruslan-mikhailov@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport release-v2.8 type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants