Tracing: stop adding HTTP headers as tracing attributes#707
Merged
Conversation
When I added the OTel tracing in #681 I went too far and added headers as tracing attributes. As @bboreham pointed out, this could log potentially sensitive headers like auth tokens. This is a quick fix to remove that from traces, we can add an allowlist or a blocklist in the future as a feature. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
colega
added a commit
that referenced
this pull request
Jun 5, 2025
OTel tracing of HTTP headers was broken, as we were using the LabelerFromContext which actually labels metrics, not adds tracing attributes. I fixed headers tracing, introducing an exclusion list similar to the one we use in logging. I also added tests for OTel tracing: I had to move these under server/internal/ because since both Jaeger and OTel tracing set up a global state, they need to run in different test packages. I took a slightly different approach for excluded headers: instead of just silently skipping them, I've added a trace attribute saying they were present in the request: that will likely make debugging easier. Ref: #707 Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Contributor
Author
|
Following up with a decent fix: #709 |
colega
added a commit
that referenced
this pull request
Jun 5, 2025
* Fix OTel tracing of HTTP headers OTel tracing of HTTP headers was broken, as we were using the LabelerFromContext which actually labels metrics, not adds tracing attributes. I fixed headers tracing, introducing an exclusion list similar to the one we use in logging. I also added tests for OTel tracing: I had to move these under server/internal/ because since both Jaeger and OTel tracing set up a global state, they need to run in different test packages. I took a slightly different approach for excluded headers: instead of just silently skipping them, I've added a trace attribute saying they were present in the request: that will likely make debugging easier. Ref: #707 Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update CHANGELOG.md Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * s/sourceIPs/source_ips/ in OTel Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * s/excludedHeadersList/excludedHeaders/ Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Sacrifice consistency, improve performance Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> --------- Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When I added the OTel tracing in #681 I went too far and added headers as tracing attributes.
As @bboreham pointed out, this could log potentially sensitive headers like auth tokens.
This is a quick fix to remove that from traces, we can add an allowlist or a blocklist in the future as a feature.
Signed-off-by: Oleg Zaytsev mail@olegzaytsev.com