Quiver expiry loss accounting includes already-resolved bundles and items

[drewrelmas]

Pre-filing checklist

• I searched existing issues and didn't find a duplicate

Component(s)

Rust OTAP dataflow (rust/otap-dataflow/)

Bug Description

When Quiver expires a partially resolved segment, its retention-loss counters include every bundle and item in the physical segment manifest. Bundles already ACKed by the subscriber are therefore reported as expired loss.

Subscriber completion is tracked per bundle in a bitmap, but expiry accounting occurs after selecting and force-completing physical segments. It calls SegmentStore::retention_drop_counts, which returns totals for the complete segment rather than only the bundles that were unresolved immediately before expiry.

segment N
├── bundle 0: 0 items, unresolved
└── bundle 1: 1 item, ACKed
                  │
                  ▼
segment reaches max_age
                  │
                  ▼
reported: 1 expired item
actual:   0 unresolved items

This can make processor.durable_buffer.loss report data loss on a healthy path even when every valid item was successfully exported and ACKed.

This issue is independent of why the remaining bundle is unresolved. A separate issue (#3891) covers one observed trigger in which the transform processor forwards fully filtered zero-item PData.

Steps to Reproduce

Add a Quiver regression scenario with one subscriber:

1. Ingest a zero-item bundle.
2. Ingest a one-item bundle into the same segment.
3. Finalize the segment.
4. Leave the zero-item bundle unresolved.
5. ACK the one-item bundle.
6. Backdate the segment beyond max_age.
7. Run cleanup_expired_segments().
8. Read retention_loss_snapshot().expired.

Equivalent segment state immediately before cleanup:

subscriber bitmap: [unresolved, resolved]
segment manifest:  [0 items, 1 item]

A second affected scenario is:

1. Create an incomplete older segment.
2. Create and fully ACK a later segment.
3. Keep the later segment physically retained behind the older incomplete cleanup boundary.
4. Age both segments beyond max_age.
5. Run expiry cleanup.

The completed later segment is also selected and charged as loss despite having no unresolved bundles.

Expected Behavior

Expiry loss describes telemetry that was still unresolved when retention forced its removal.

For the partially resolved segment:

expired segments = 1
expired bundles  = 1
expired items    = 0

For a fully resolved later segment retained behind an older gap, the later segment contributes no expired bundles or items.

Actual Behavior

Quiver reports complete manifest totals for each selected segment:

expired segments = 1
expired bundles  = 2
expired items    = 1

The one reported item belongs to the already-ACKed bundle. The actual unresolved bundle contains zero items.

More generally, an ACKed bundle's item count is added to expired loss whenever its physical segment is selected for retention expiry.

OTel-Arrow Version

latest

Environment

N/A

Configuration

No collector configuration is required for the deterministic Quiver regression. Configure Quiver with a short `max_age`, place the unresolved and resolved bundles in the same finalized segment, and invoke `cleanup_expired_segments()` after backdating the segment.

The integration topology in which this was observed was:


receiver -> transform -> durable buffer -> exporter

Log Output

The deployed segment state was:


segment 1
├── bundle 0: 0 items, repeatedly NACKed
└── bundle 1: 1 item, successfully exported and ACKed

maintenance result:
  persistence_dropped_items{reason="expired"} += 1

actual unresolved valid items:
  0


All valid records in the test cohort were successfully exported. The unresolved bundle was an additional zero-item message, so the reported expired item came entirely from acknowledged data.

Additional Context

No response