feat: scope backfill-by-copy and removal reconciliation jobs

[VVoruganti]

Summary

Adds the retroactive-membership half of the Scopes RFC (DEV-1970): when a session with pre-existing messages is added to a scope, its history is backfilled into the scope's memory; when a session is removed, the scope's copies are reconciled away.

• Backfill-by-copy, zero LLM re-derivation. Explicit-level documents are session-pure (the DEV-2000 invariant merged into this branch), so a session's explicit conclusions are identical whether observed by the sender or by any observer. Adding a session to a scope after the fact is therefore pure row-copying — no re-running the deriver — from each observed peer's global (P, P) collection into the scope's (scope_peer, P) collection. The only external call is an embedding lookup for rows whose embedding is NULL (external vector store deployments); never an LLM call.
• Idempotency via copied_from. Each copy stamps internal_metadata.copied_from = <source doc id>. Re-processing the same backfill is a no-op; a soft-deleted copy left by an earlier removal is restored rather than duplicated, so add → remove → re-add converges on exactly one live copy per source document.
• Removal cascade + rebuild. Removing a session soft-deletes its explicit copies in the scope's collections, then fail-closed cascades the soft-delete transitively to derived documents whose source_ids intersect anything removed (a deduction resting on removed evidence leaves with it, and so does an induction resting on that deduction). Enqueues a card_refresh dream with rebuild=True (drops the stale card from the prompt so it's rebuilt from remaining evidence only) plus a manual omni dream to rebuild the higher-order layer.
• Status surface. Per-session backfill job state (pending / completed / failed, plus docs_copied once complete) lives in the scope peer's internal_metadata, written via single-statement JSONB merges (concurrent-writer safe), and served by GET /v3/workspaces/{w}/scopes/{scope_id}/status.
• Enqueue wiring. Adding a session to a scope (POST /scopes/{scope_id}/sessions) or creating a session with scopes set enqueues scope_backfill only when the session already has messages; removal (DELETE /scopes/{scope_id}/sessions/{id}) always enqueues scope_removal.

Bug fixed while writing tests

update_scope_backfill_status (src/crud/scope.py) passed json.dumps(...)'d strings through SQLAlchemy's cast(..., JSONB). That double-encodes: psycopg's JSONB bind adapter re-serializes the already-JSON string, producing a JSONB string scalar instead of an object. Postgres's || between two non-array jsonb scalars doesn't merge keys — it silently wraps both sides into a 2-element array. This corrupted backfill_status into a list, and later crashed clear_scope_backfill_status's #- path-delete (called on every removal) with InvalidTextRepresentation: path element at position 2 is not an integer, since Postgres saw an array where it expected an object. In other words: removal reconciliation broke for any session that had ever completed a backfill. Fixed by passing raw Python dicts to cast() instead of pre-serialized JSON strings, mirroring the already-correct pattern in update_collection_internal_metadata in the same file.

Also added src.deriver.scope_backfill.tracked_db to the tracked_db patch allowlist in tests/conftest.py — the new module was missing from that per-import-site list (every module that does from src.dependencies import tracked_db needs its own entry), so its DB work was silently running against the real configured database instead of the isolated per-test database.

Tests

tests/deriver/test_scope_backfill.py (9 tests, all passing):

1. Backfill copies exactly the target session's explicit docs (not other sessions', not derived levels), with copied_from set and embeddings carried over.
2. Idempotency: same backfill processed twice → no duplicates; add → remove → re-add → docs live exactly once.
3. Multi-peer session: docs copied per observed peer into the right (scope, P) collections.
4. Removal cascade: explicit copies soft-deleted; a derived doc whose source_ids references a removed copy is soft-deleted; an unrelated derived doc survives.
5. Dream enqueues: backfill completion enqueues a manual omni dream; removal enqueues card_refresh with rebuild=True plus omni (asserted via QueueItem payloads, no LLM calls).
6. Status endpoint reflects pending → completed with docs_copied.
7. Route wiring: adding a session with messages enqueues backfill; an empty session does not.

Full non-integration/SDK/bench/live-llm suite: 1339 passed, 3 pre-existing failures unrelated to this change (tests/crud/test_document.py semantic-dedup tests require an OpenAI embedding key).

Fixes DEV-1999 (part of DEV-1970 Scopes RFC)

Test plan

• uv run pytest tests/deriver/test_scope_backfill.py — 9/9 passing
• uv run pytest tests/ (excluding integration/sdk/bench/live_llm) — 1339 passed, 3 known pre-existing failures unrelated to this change
• uv run basedpyright — 0 errors, 0 warnings
• uv run ruff check / ruff format — clean on all touched files

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a48dce32-0dd9-4390-a043-ae5536b5dce5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch vineeth/dev-1999

---

_{Comment @coderabbitai help to get the list of available commands.}