Patch WAL dedupe when search is disabled

[annanay25]

What this PR does:
Fixes an issue with WAL dedupe (SearchDataCombiner) when search is disabled. This happens because we still create search WAL files on the ingester with empty entries.

Which issue(s) this PR fixes:
Fixes #967. Part of #932

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

This happens because we still create search WAL files on the ingester with empty entries.

I'm fine with this fix, but should we also not write an empty entry? Doesn't that mean we're incurring some write penalties even with search disabled?

[annanay25]

I'm fine with this fix, but should we also not write an empty entry? Doesn't that mean we're incurring some write penalties even with search disabled?

So we are basically writing zero length objects to the search data file, which I don't think is costly. The solution to this is to have

if i.searchEnabled {
  // bail
}

sprinkled all over the ingester code (CutCompleteTraces, CompleteBlock, Flush, ... etc).

Not sure if we want that, but open to suggestions.