Backend Search

[joe-elliott]

What this PR does:
Adds support for full backend search to the /api/search endpoint. If parameters start and end are provided Tempo will perform a full backend search by scanning the blocks in parallel. This PR also includes code to run and configuration to use a Google Cloud Function to perform the scanning of blocks.

Changes:

• Additional query-frontend params related to search. default_result_limit and max_result_limit were moved from the querier config to here:

query_frontend:
  search:
    concurrent_jobs: <int>
    target_bytes_per_job: <int>
    default_result_limit: <int>
    max_result_limit: <int>
    max_duration: <duration>
    query_ingesters_within_min: <duration>
    query_ingesters_within_max: <duration>

• Additional storage parameters related to search:

storage:
  trace:
    search:
      chunk_size_bytes: <int>
      prefetch_trace_count: <int>

• Full Backend and Serverless integration tests
• Fixes on some flakey integration tests by using cortex_e2e.WaitMissingMetrics
• Testing on model.Matches()

Which issue(s) this PR fixes:
Fixes #1112

Even though we are clearing #1112 as part of this PR backend search is far from complete. A new issue will be created "Backend search production readiness" (#1175) that will track future search work.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yvrhdn]

Really nice work consolidating all those search functions!

[mdisibio]

Excellent work! LGTM