Retention deletion does not work on S3 or GCS when a prefix is configured

[bpfoster]

Describe the bug
When object storage is configured to use AWS S3, and a prefix is configured, if retention is configured it does not appear to delete old data beyond the retention period.

To Reproduce
Steps to reproduce the behavior:

1. Start Tempo v2.4.0
2. Wait for blocklist_poll time to elapse.
3. Wait for compacted_block_retention time to elapse.
4. Wait a lot longer.
5. Witness no files deleted from S3.

Expected behavior
I expect data files in S3 older than the configured block_retention to be deleted from S3.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

Additional Context
I see 0 blocks listed in the compactor logs:

level=debug ts=2024-03-05T17:03:31.370043415Z caller=s3.go:273 msg="listing blocks" keypath=foo/tempo/ found=1 IsTruncated=false NextMarker=
level=debug ts=2024-03-05T17:03:32.513352722Z caller=s3.go:389 msg="listing blocks complete" blockIDs=0 compactedBlockIDs=0

Notable configuration:

compactor:
  compaction:
    block_retention: 48h
    compacted_block_retention: 10m
storage:
  trace:
    backend: s3
    blocklist_poll: 1m
    s3:
      bucket: foobar-bucket
      prefix: foo/tempo

When I have no prefix configured, I see a lot of compaction/retention activity in the (debug) log. With a prefix configured, I see virtually nothing.

[joe-elliott]

Does this issue occur in 2.3.1? or is it new in 2.4.0? We did make some changes to the polling logic and I wonder if that's the root cause. Is Tempo working otherwise? Traces are being returned correctly from object storage?

Retention logic is here:
https://github.com/grafana/tempo/blob/main/tempodb/retention.go

I don't believe much, if anything, changed here and I don't see anything that stands out as impacted by a s3 prefix.

[bpfoster]

Hi @joe-elliott . I don't have an answer on whether it occurred in 2.3.1. I'm setting up a new installation for the first time so v2.4.0 is my only experience. As far as I can tell, everything otherwise is working as expected.

I'm quite unfamiliar with the tempo codebase. I found one potentially problematic area in tempodb/backend/s3/s3.go. Please see PR #3466.
On debugging, I noticed that all objects from S3 were filtered out in the affected method, leaving with no results. I can't say for sure this is the only (or best!) fix, but running it locally for a brief period of time results in the expected cleanup behaviors.