[bugfix] correct lag calculation

[ruslan-mikhailov]

What this PR does: fixes lag calculation. This PR contains couple of bugfixes:

1. correct lag calculation as diff between HWM and latest committed offset.
   a. If there was an error on consumption right after the first iteration, we cannot get lag and set it to -1. Note: current code does not return an error, it is a no-op change.
   b. if no error and no recorded offset - means no new records, lag is zero.
2. record lag only after consume function call
3. If all records in a batch could not be consumed (e.g. record.Timestamp.Before(cutoff)), we still need to commit these changes. Currently, they are committed only if at least one message was pushed to inst.pushBytes.
4. Wait for catch up only after all processes started. This fixes a deadlock when live-store is not yet started block processing, while it is already consuming and blocked by waitForCatchUp.

How it has been tested: unit tests (in the PR) and manual tests.
Manual test case.

1. add long sleep on consumption to be sure live-store is lagging.
2. run it for at least 10 minutes (ideally, 30+ mins)
3. remove long sleep and enable readiness probe.

Expected result: it is running, but in non-ready state. After it catches up, it becomes ready.

Actual result matches expected results:

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[oleg-kozlyuk-grafana]

Excellent job, thank you for fixing these issues!