Clone the diff in GRPC streaming to prevent panics

[joe-elliott]

What this PR does:
Clones the diff in GRPC streaming to prevent panics

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[javiermolinar]

Should not this mutex prevent that? the lock is adquired on diff and grpc_final:

tempo/modules/frontend/combiner/common.go

Line 107 in f0b76c4

c.mu.Lock()

[joe-elliott]

Should not this mutex prevent that? the lock is adquired on diff and grpc_final:

The issue is that the pointers passed out of these functions are eventually passed to the proto marshalling code. These same pointers are also held inside the combiner and modified. The issue occurred when one of the objects was modified as it was being marshalled (we think).

[github-actions]

The backport to r160 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-3961-to-r160 origin/r160
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 89b8d7d783e7e4c8b2955bd3b39c9e084f75ca4c

When the conflicts are resolved, stage and commit the changes:

git add . && git cherry-pick --continue

If you have the GitHub CLI installed:

# Push the branch to GitHub:
git push --set-upstream origin backport-3961-to-r160
# Create the PR body template
PR_BODY=$(gh pr view 3961 --json body --template 'Backport 89b8d7d783e7e4c8b2955bd3b39c9e084f75ca4c from #3961{{ "\n\n---\n\n" }}{{ index . "body" }}')
# Create the PR on GitHub
echo "${PR_BODY}" | gh pr create --title '[r160] Clone the diff in GRPC streaming to prevent panics' --body-file - --label 'type/bug' --label 'backport' --base r160 --milestone r160 --web

Or, if you don't have the GitHub CLI installed (we recommend you install it!):

# Push the branch to GitHub:
git push --set-upstream origin backport-3961-to-r160

# Create a pull request where the `base` branch is `r160` and the `compare`/`head` branch is `backport-3961-to-r160`.

# Remove the local backport branch
git switch main
git branch -D backport-3961-to-r160

[github-actions]

The backport to r161 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-3961-to-r161 origin/r161
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 89b8d7d783e7e4c8b2955bd3b39c9e084f75ca4c

When the conflicts are resolved, stage and commit the changes:

git add . && git cherry-pick --continue

If you have the GitHub CLI installed:

# Push the branch to GitHub:
git push --set-upstream origin backport-3961-to-r161
# Create the PR body template
PR_BODY=$(gh pr view 3961 --json body --template 'Backport 89b8d7d783e7e4c8b2955bd3b39c9e084f75ca4c from #3961{{ "\n\n---\n\n" }}{{ index . "body" }}')
# Create the PR on GitHub
echo "${PR_BODY}" | gh pr create --title '[r161] Clone the diff in GRPC streaming to prevent panics' --body-file - --label 'type/bug' --label 'backport' --base r161 --milestone r161 --web

Or, if you don't have the GitHub CLI installed (we recommend you install it!):

# Push the branch to GitHub:
git push --set-upstream origin backport-3961-to-r161

# Create a pull request where the `base` branch is `r161` and the `compare`/`head` branch is `backport-3961-to-r161`.

# Remove the local backport branch
git switch main
git branch -D backport-3961-to-r161