Implement generic forwarder

[Blinkuu]

What this PR does:

This PR introduces a new subcomponent to distributor named forwarder. This generic forwarder serves as a facade for forwarding batches of spans in "fire-and-forget" way. The feature is designed in such a way that it's configurable on per-tenant basis.

Example configuration:

# config.yaml
distributor:
  forwarders:
    - name: "otel-forwarder"
      backend: "otlpgrpc"
      otlpgrpc:
        endpoints: ['otelcol:4317']
        tls:
          insecure: true

# overrides.yaml
overrides:
  "example-tenant-1":
    forwarders: ['otel-forwarder']
  "example-tenant-2":
    forwarders: ['otel-forwarder']

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[CLAassistant]

All committers have signed the CLA.

[mapno]

Gave it a quick review. I will come back to it later, it's a big PR. Overall I like the direction of the PR.

[mapno]

Overall, looks very good to me. I only find the manager logic to be complicated and difficult to follow. Left a comment.

Will wait to see if anyone else wants to chip in.

[joe-elliott]

This is a large PR that will require several hours to give a proper review to. Is there anyway to break the refactor up from the functionality addition?

[Blinkuu]

This PR is now rebased on top of #1796. That PR must be merged before this one.

[joe-elliott]

I think we can reduce the complexity of this PR significantly. Let me know if some of these comments are unclear.

[mapno]

I think it's almost there. Great work 👏

[joe-elliott]

One final question and then I'm good.

[mapno]

LGTM