Query-frontend: Performance Improvements and Refactor

[joe-elliott]

What this PR does:
Refactors the frontend to pass a Request interface instead of an *http.Request. This is progress toward supporting deterministic search. Also added a search pipeline benchmark and found a perf improvement in prepareRequestForQueriers.

> benchstat before.txt after.txt
goos: darwin
goarch: arm64
pkg: github.com/grafana/tempo/modules/frontend
                  │  before.txt  │             after.txt              │
                  │    sec/op    │    sec/op     vs base              │
SearchPipeline-11   609.4m ± ∞ ¹   604.0m ± ∞ ¹  -0.87% (p=0.008 n=5)
¹ need >= 6 samples for confidence interval at level 0.95

                  │   before.txt   │              after.txt               │
                  │      B/op      │     B/op       vs base               │
SearchPipeline-11   10.642Mi ± ∞ ¹   8.659Mi ± ∞ ¹  -18.63% (p=0.008 n=5)
¹ need >= 6 samples for confidence interval at level 0.95

                  │  before.txt  │             after.txt              │
                  │  allocs/op   │  allocs/op    vs base              │
SearchPipeline-11   163.0k ± ∞ ¹   148.3k ± ∞ ¹  -9.02% (p=0.008 n=5)
¹ need >= 6 samples for confidence interval at level 0.95

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[mapno]

Refactor looks OK. Any idea why this is faster or uses less memory?

[javiermolinar]

I find this function pretty obscure. Why don't we return a new http.Request based on the original one?

func getQuerierHTTPRequest(origin_req *http.Request, tenant string) *http.Request{
}

That way we will reduce the cognitive overload reading the code

[joe-elliott]

I would prefer modifying in place to reduce allocs

[joe-elliott]

Refactor looks OK. Any idea why this is faster or uses less memory?

I removed these calls to .Query() which allocs a map by changing the way prepareRequestForQueriers works:

https://github.com/grafana/tempo/pull/3888/files#diff-85974350e62b77bee6b5d88d537e3946839ccb1bba5f6dc89450a1c5489b5d4fL443

I have a follow up PR coming that will remove millions of these calls from the search path.