Search stability improvements

[mdisibio]

What this PR does:
This PR fixes several rough spots:

• Fixes a mutex deadlock that can occur between search and instance.writeTraceToHeadBlock, by taking i.blocksMtx until all search routines are started in instance.Search
• Fixes panics channel already closed and send on closed channel in search.Results by redoing the cleanup and using sync.WaitGroup
• Improves overall search contention by using read-locks where possible
• Fix wal search error "file already closed" where the wal block is deleted after search is initiated but before the block is opened, by tracking closed state in the block.
• Fix flakey test by avoiding t.TempDir, behavior discussed here: golang/go/issues/43547
• Fix flakey data race by guarding StreamingSearchBlock.FlushBuffer with a mutex

Which issue(s) this PR fixes:
Fixes n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[mapno]

lgtm. probably another couple of eyes would be good

[annanay25]

Really nice batch of changes, just one comment on trying to understand the deadlock state better.

[annanay25]

Can add a bunch of bugfixes to the Changelog :)

[joe-elliott]

Looks good to me, but we are getting a test timeout in integration tests.

panic: test timed out after 10m0s

[mdisibio]

The test that timed out is TestScalableSingleBinary. This test pushes 1 trace and is somewhat new. Wondering if it is related or if the test is flakey. Will review.

[zalegrala]

If the e2e tests are timing out, this might have to do with waiting for the metric to show up in the /metrics endpoint.

[mdisibio]

Unable to reproduce the flakey tests locally, and consensus is they are likely not caused by these changes. Merging.