Skip to content

fix(storage): bound projection retries and cut v0.40.12 - #535

Open
raphaelmansuy wants to merge 25 commits into
edgequake-mainfrom
cursor/review-pr534-projection-retries-8d08
Open

fix(storage): bound projection retries and cut v0.40.12#535
raphaelmansuy wants to merge 25 commits into
edgequake-mainfrom
cursor/review-pr534-projection-retries-8d08

Conversation

@raphaelmansuy

@raphaelmansuy raphaelmansuy commented Nov 8, 2026 •
edited by cursor Bot
Loading

Copy link
Copy Markdown
Owner

Summary

Lands the projection retry ceiling from #534 on top of the published v0.40.11 line and cuts v0.40.12. Merging this fast-forwards edgequake-main through the already-shipped v0.40.8–v0.40.11 commits (schema 231). Squash-merging would drop that schema and must not be used.

#534 itself stays on the contributor fork, which refuses a push that updates .github/workflows (the app token has no workflows permission). Rebasing that head onto v0.40.11 is therefore impossible from here. Merging #534 as it stands would base the fix on v0.40.7 (schema 227) and regress every release since v0.40.8.

The fix

projection_deliveries.attempts increments when a row is claimed. The worker used to retry a failing apply forever, which is how a dense document (#511) loops until the process runs out of memory.

  • Default budget is 10 claims (EDGEQUAKE_PROJECTION_MAX_ATTEMPTS). 0 keeps the old unbounded behavior. Unset or unparsable keeps the default.
  • The last allowed attempt still gets its provider call. A claim already past the ceiling is quarantined with no provider call.
  • A failed one-delivery batch is quarantined once. It is not applied a second time.
  • A failed multi-delivery batch is re-applied one item at a time, so one exhausted delivery does not take its siblings with it.
  • An ack failure after a successful apply still releases for retry. A visibility blip does not dead-letter work that already succeeded.
  • Quarantine writes the reason into projection_deliveries.receipt (bytea). Recovery is a SQL reset to pending with attempts = 0. A quarantined document_batch delivery keeps that document's serving fence closed.

#522 (duplicate endpoint ids) is a separate change and is not in this cut.

Release

  • Version 0.40.12. Schema stays 231. edgequake migrate is a no-op.
  • No new epoch row. v0.40.8 already covers migration-tree hash ba0c9db9d18d39c1.
  • SPEC-001 mix retrieval is unchanged, so Acc was not re-scored. Pack medical-mid-20261024T031725Z stays the attestation: EdgeQuake 0.8084, LightRAG 0.7681, delta +0.0403, 95% CI [-0.0241, +0.1036], n_paired 72, a statistical tie.

CI

edc40a9ed fixed the three gates this cut exposed:

  • postgres-roles adds the google_project column the saved-connection insert writes.
  • Security Audit ignores RUSTSEC-2026-0429, the same ZipArchive::extract abort already ignored as 0367/0368/0369. edgeparse-core 0.3.2 requires zip 2 and only writes archives. utoipa-swagger-ui 9 uses zip 3 at build time and copies the upstream Swagger UI archive. Neither calls extract() on untrusted input.
  • E2E UI Gate runs on ubuntu-24.04 because Playwright 1.58 has no Chromium build for ubuntu26.04-x64.

That run then executed the suite (390 passed) and failed six tests that v0.40.11 never reached, because its install died on Ubuntu 26.04. 82e24ee91 aligns those tests with the screens:

  • The OpenAI key field is labeled API key (optional) unless the shape sets requires_key.
  • An untested fleet provider shows why models are locked and does not render a model id box.
  • Saves post to /connections?scope=deployment, so the tenant-header assertion has to match that URL.
  • The picture suggestion is read after the corpus query resolves.

5b9ad3bba stops the compile jobs dying on pdfium. pdfium-auto 0.3.1 downloads chromium%2F7961. GitHub answers that URL, and the slash-form HTML download, with HTTP 504, and the build script aborts. CI and the release image now fetch the chromium/7961 asset through the release API and set PDFIUM_BUNDLE_LIB.

Layer 1 on pull_request_target still fails, and this pull request cannot make that job green. That event checks out edgequake-main (v0.40.7, schema 227), so tags v0.40.8–v0.40.11 look uncovered. The same Layer 1 job on the pull_request event, which checks out this head, passed. The pull_request_target Layer 2 job also compiles that stale base, so it can still die on the same pdfium 504 until the merge updates the base. Merging repairs both.

Verification

  • cargo test -p edgequake-storage --features postgres --lib --locked — 632 passed
  • OpenAPI snapshot contract test passed
  • cargo fmt --check and storage clippy -D warnings clean
  • Epoch coverage 141/141 on this head, schema train 231, migration checksums, env registry
  • Prefetch script downloaded libpdfium.so (ELF x86-64) through the release-asset API from this environment
  • Docker, Postgres e2e, and a fresh Acc score were not run in this environment
Open in Web Open in Cursor 

cursoragent and others added 18 commits November 7, 2026 04:31
@cursoragent @raphaelmansuy
perf(db): plan the clearance policy once and cover its columns
72acfe0 
Migration 220 put a restrictive clearance predicate on eight tables. Two
terms of that predicate are the same value for the whole statement (break
glass, and whether the workspace is enforcing), but they sat inside the
per-row clearance_covers() function, so the planner called the function for
every candidate row and a scalar subquery could not be hoisted out of it.

Split the predicate where row-independence ends: clearance_bypass() holds
the row-independent half, clearance_rank_covers() keeps rank and compartment
coverage, and clearance_covers() stays as bypass OR rank_covers so the
out-of-policy callers in graph/plain keep working unchanged. Policies call
the bypass as (SELECT clearance_bypass()), which the planner turns into an
InitPlan — the same technique migration 208 applied to current_tenant_id().

Migration 229 gives the (tenant_id, workspace_id) indexes on chunks and
documents the clearance columns as INCLUDE payload, so the policy is
answered from the index instead of a heap fetch per candidate row. Each
index replaces the existing one, so no table gains an index to maintain.
Two byte-identical duplicate index pairs are dropped.

Measured as edgequake_tenant_access via SET LOCAL ROLE (the path
rls::with_rls_transaction takes), 120k chunks / 36k in one workspace:

  chunk count           598 ms -> 6 ms
  chunk FTS page        267 ms -> 107 ms
  document page          13 ms -> 0.9 ms
  document count         12 ms -> 0.3 ms

Equivalence is proven exhaustively over every combination of enforcement,
break-glass, rank, allow-empty, compartments and workspace settings:
19,440 comparisons, 0 mismatches, against the verbatim migration-220
definition. e2e_spec228 additionally fails if the bypass is not hoisted.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
test(db): cover the clearance InitPlan split and its equivalence
fecfded 
Adds a live-Postgres e2e test that compares the split predicate against
clearance_covers() across every combination of enforcement, break-glass,
rank, allow-empty, compartments and workspace settings, and asserts the
planner hoists the bypass (InitPlan `$n` before the per-row `SubPlan`) and
reaches the chunks scope index as an index-only scan.

Keeps the HEAD schema train contracts current: 229 is the head, the
release map covers 226 through 229, and two new contracts pin the shape of
migrations 228 and 229.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
perf(db): finish the RLS InitPlan rewrite for six late policies
ddf611a 
Migration 208 wrapped session helpers in `(SELECT fn())` inside policy
expressions and documented the invariant: "New policies must use the
wrapped form ... e2e_rls_initplan_policies (edgequake-storage) fails if any
are left bare." That guard test was never written, and five later migrations
reintroduced the bare form — 215, 216, 219, 222 and 225 — so six policies
still call the helpers once per row.

A bare STABLE helper in a policy becomes a Filter the executor re-evaluates
for every emitted row. The plans show it: the index-only scan carries
`Filter: $0` over 4,031 rows on workspace_term_stats and 372,000 on
chunk_terms. Wrapped, both scans carry no filter and the helper is a single
InitPlan.

Measured as edgequake_tenant_access after SET LOCAL ROLE, 120k chunks and
2.2M chunk_terms rows, median of 9 alternating runs on an otherwise
identical pair of databases:

  full BM25 statement   24.6 ms -> 12.4 ms  (9 of 9 runs improved)

The composite statement gains what an isolated count(*) does not, because a
count reads only the index while BM25 joins both tables and re-scans
chunk_terms three times.

Meaning is unchanged: app.current_* are fixed for the statement, so the
wrapped call returns what the bare call returned. Verified on the three
states that could differ — unset, set and malformed GUC — and the rewrite
aborts if unwrapping does not reproduce the original expression.

Adds the guard test 208 promised, so this cannot rot again.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
test(db): add the missing RLS InitPlan guard and bump the train to 230
bcdbcc9 
e2e_rls_initplan_policies fails when any policy expression still calls a
session helper bare, naming the table, policy and clause. It fails on a
database before migration 230 and passes after — verified both directions.

Also pins 230 in the HEAD train contracts and the release map, and records
the fix in docs/migrations.md.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
test(storage): realign six stale source-text contracts
db2ffa8 
Six contract tests asserted source-text literals that the code has since
outgrown. All six failed on edgequake-main before this branch existed, so
they were dead weight rather than a signal; each was confirmed unrelated to
the migrations here (no production source changed in this commit).

Each is realigned to the design now in place, keeping the invariant the test
existed to defend:

- incident edges: assert the OR template that replaced "OR ag_catalog"; the
  EDGE child table and no-UNION rules are unchanged.
- hot node reads: IMP-031 moved node reads off Cypher property MATCH onto
  native SQL (cypher_query_bound is now fallback-only); assert that.
- eq_* DDL early exit: assert the O(1) catalog probe short-circuits before
  any DDL, rather than one reworded log line.
- eq_* sync triggers: they moved to helpers::sync_triggers and use CREATE OR
  REPLACE, so assert no DROP TRIGGER on the hot path instead.
- discovery: migration 207 moved the GIN-eligible probe into a SECURITY
  DEFINER lookup; assert ?| plus the unnest/generate_series probe CTE there.
- vector DDL GUCs and PreferExisting: both were refactored into helpers and
  the provider; assert the behaviour at its new home.

edgequake-storage now passes 180/180 suites, up from 173 with 6 failing.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
fix(ci): repin the unreleased epoch to schema 230
8cd5e4e 
Migration Guard / Layer 1 failed on this branch: check_epoch_coverage.sh
compares the [unreleased] pin against HEAD's numbered-migration set, and the
pin still described schema 227 / 225 files while HEAD is 230 / 228.

  unreleased pin 7491afd4e3a729e7 != HEAD db2868e7660eaf55 (max=230 files=228)

Repin to the measured HEAD hash. `./scripts/check_epoch_coverage.sh` now
reports "HEAD schema set: max=230 n=228 hash=db2868e7660eaf55 [OK unreleased]"
and every published tag still resolves to an epoch.

Nothing here is published-tag metadata: no [[epoch]] entry changes, so the
v0.40.x history keeps its own hash.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
docs: advertise schema train 230 in the upgrade guide and notes
754df20 
Migration Guard / Layer 1's "Schema train docs parity" step failed: the
check compares the highest numbered migration (230) against manifest, then
requires the number to appear in docs/operations/upgrading.md,
edgequake/docs/migrations.md and edgequake/migrations/NOTES.md.

upgrading.md and NOTES.md still stopped at 227, and the per-migration table
in upgrading.md had no rows for 228-230. Adds the three rows and refreshes
both train headers. `./scripts/check_schema_train_docs.sh` passes.

Also states plainly that 228-230 are unreleased — no [[epoch]] entry is
added here, so the v0.40.x published-tag history keeps its own hash.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
release: bump to v0.40.8
ba38afc 
A Vertex connection saves its Google Cloud project. Queries, chat, and
gemini-embedding-2 use that project, so a server without
GOOGLE_CLOUD_PROJECT can still answer. Schema moves 227 to 231
(migrations 228-231). Row-security InitPlans from 228-230 ship in this cut.

SPEC-001 Acc is not re-scored. Mix retrieval is unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
release: bump to v0.40.9
35cc6e4 
Deployment Providers can edit a provider. Update saves the name, base URL,
key, and the Google Cloud project for Vertex. Schema stays 231.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
release: bump to v0.40.10
3812ed4 
The v0.40.9 image publish was cancelled when quality-gates hit 90 minutes
on a cold compile. Raise that job limit to 180 minutes so the same
provider Edit can publish. Schema stays 231.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
fix(ui): choose models by the job they can do
9b8da40 
The organization ceiling and the workspace list were a grid of empty
boxes. An empty box meant the job was open, which reads as off. Each
model now lists only the jobs it can do, grouped by provider, and a job
stays open until the first model is allowed.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
release: v0.40.11 — query uses the saved Vertex project
926d357 
A Normal or image query built gemini-embedding-2 from the process
environment and ignored the project stored on the Vertex connection.
Query, chat, and image ingest now open that connection. Schema stays 231.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@dajiaohuang @cursoragent
fix(storage): bound projection retries
97160a0
@cursoragent @raphaelmansuy
fix(storage): bound projection retries without doubling failed calls
a60e56d 
Carry forward the retry ceiling from #534 and close the gaps a review
found:

- The last allowed attempt (attempt == max) was quarantined before its
  apply call, so a delivery that would succeed on attempt max was retired
  without ever being tried. Split the two boundaries: over budget
  (attempt > max) quarantines on sight, exhausted (attempt >= max)
  quarantines only after a failure.
- A one-delivery batch re-applied its exhausted item after the batch call
  had already failed. That pays the same expensive failing provider call
  (the timeout/OOM path in #511) twice for no discriminating value. Only
  multi-delivery batches isolate now.
- Move the retry policy into small helpers (attempts_over_budget,
  attempts_exhausted, backoff_ms, renew_for, count_apply_call,
  release_retry, quarantine_exhausted, retry_failed_batch,
  isolate_exhausted) instead of one 130-line match arm.
- Add EDGEQUAKE_PROJECTION_MAX_ATTEMPTS so operators can tune or disable
  the ceiling (0 keeps the pre-fix unbounded retries).

Tests cover the ceiling boundaries, batch isolation keeping a healthy
sibling alive, unbounded mode, and env parsing.

Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
docs: document the projection retry ceiling and quarantine recovery
fd54fa9 
Operators need to know a delivery now stops at 10 claims (0 disables the
ceiling) and how to re-drive one after fixing the cause: the reason lives
in projection_deliveries.receipt, and resetting attempts is the recovery.
Register EDGEQUAKE_PROJECTION_MAX_ATTEMPTS so the env gates stay honest.

Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
docs: state what the projection attempt counter does and does not bound
3e58ec0 
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
docs: quarantine does not block the serving fence
aa292fa 
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
release: bump to v0.40.12
ba77607 
Projection deliveries stop after 10 claimed attempts. A failed batch
quarantines only the delivery that still fails on its own, and a
one-delivery batch does not pay that call twice. Schema stays 231, so
edgequake migrate is a no-op on a database already at 231.

SPEC-001 Acc is not re-scored. Mix retrieval is unchanged.

Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursor cursor Bot mentioned this pull request Nov 8, 2026
Open
cursoragent and others added 7 commits November 8, 2026 07:54
@cursoragent @raphaelmansuy
fix(ci): unblock v0.40.12 gates that the projection cut exposed
edc40a9 
The saved-connection probe inserts google_project, so the SPEC-163
role database has to add that column. cargo audit now ignores
RUSTSEC-2026-0429, the same ZipArchive::extract abort already accepted
for zip 2 and zip 3. The UI gate stays on Ubuntu 24.04 because
Playwright 1.58 has no chromium build for ubuntu-latest's 26.04 image.

Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
test(webui): align the UI gate with the current connection and query …
82e24ee 
…screens

The key field is labeled optional unless the shape requires one. An
untested fleet provider explains itself and does not offer a model id.
Connection saves go to /connections?scope=deployment, and the picture
suggestion is read after the corpus query resolves.

Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
fix(ci): prefetch pdfium through the release asset API
5b9ad3b 
pdfium-auto 0.3.1 downloads chromium%2F7961. GitHub answers that URL,
and the slash-form HTML download, with HTTP 504, and the build script
aborts. CI and the release image now fetch the chromium/7961 asset
through the release API and set PDFIUM_BUNDLE_LIB.

Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
fix(ci): install bun through the release asset API
cb2a8f6 
oven-sh/setup-bun downloads the HTML release URL, which returns HTTP 504.
The UI gate never calls bun, so that step is removed. Jobs that run bun
install fetch the latest release through the asset API.

Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
fix(ci): put bun and bunx on PATH for the UI and release gates
f590d25 
Playwright starts the UI gate with bun run, and release_gates.sh typechecks
with bunx. The asset install only copied the bun binary, so bunx was missing
and the UI gate had no bun at all.

Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
fix(ci): fail closed when pdfium or Swagger UI cannot be fetched
555724d 
A workflow token is rejected with HTTP 403 on another repo's release
asset, and echo "VAR=$(download)" hides that failure and exports an
empty path. pdfium is fetched without the token. Swagger UI is fetched
from codeload, which serves the tag zip the github.com archive URL
answers 504 for. A failed download now fails the step.

Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursoragent @raphaelmansuy
fix(ci): vendor pdfium and Swagger UI so builds do not download them
f8d253a 
GitHub answers the pdfium asset API and the Swagger UI archive with
HTTP 502, 503, and 504. CI and the release image now use the
chromium/7961 linux archives and Swagger UI v5.17.14 checked into the tree.

Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@raphaelmansuy @cursoragent @dajiaohuang