Skip to content

release: v0.40.11 β€” query uses the saved Vertex project - #533

Open
raphaelmansuy wants to merge 12 commits into
edgequake-mainfrom
cursor/postgres-query-index-audit-7367
Open

raphaelmansuy wants to merge 12 commits into
edgequake-mainfrom
cursor/postgres-query-index-audit-7367

Conversation

@raphaelmansuy

@raphaelmansuy raphaelmansuy commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Description

v0.40.11 is tagged (v0.40.11 β†’ 926d357b6). Schema stays 231. Upgrade notes: docs/operations/upgrade-to-0.40.11.md.

A query on a Normal or image workspace still failed with GOOGLE_CLOUD_PROJECT is not set after the project was saved on the Vertex card. The query built gemini-embedding-2 from the process environment and never opened the connection. Query, chat, and image ingest now use the project saved on that connection. A stored service account is used when one is saved. The image publish is Release β€” Docker (GHCR). Deploy GCP rolls the demo after that workflow succeeds, because the tag branch is v0.40.11.

The organization model ceiling and the workspace model list, arranged by job, ship in this same image.

v0.40.10 is tagged (v0.40.10 β†’ 3812ed443). Schema stays 231. Upgrade notes: docs/operations/upgrade-to-0.40.10.md.

The v0.40.9 image publish (Release β€” Docker) was cancelled when quality-gates hit its 90 minute limit during a cold compile. The tests had passed. v0.40.10 raises that limit to 180 minutes. The new publish is Release β€” Docker (GHCR). Deploy GCP rolls the demo after that workflow succeeds.

Edit a provider

Deployment β€Ί Providers had Refresh and Remove, and no way to change a provider. Each card now has Edit. Update saves the name, base URL, a replacement key, and, for Vertex, the Google Cloud project. Leave the key blank to keep the key the server already has. The demo picks this up when the v0.40.10 image publish finishes and Deploy GCP rolls it.

v0.40.8 is tagged (v0.40.8 β†’ ba38afcb2). Schema moves 227 β†’ 231. Run edgequake migrate before the new API replicas. Upgrade notes: docs/operations/upgrade-to-0.40.8.md.

The image publish for v0.40.8 is Release β€” Docker (GHCR). v0.40.9 was cancelled at the 90 minute gate (run). v0.40.10 is Release β€” Docker (GHCR).

Model ceiling, by job

The organization ceiling was a grid of empty boxes. An empty box meant the job was open, so a long list of chat models looked like nothing was allowed, and a dash hid the difference between "cannot do this job" and "not chosen."

Each model now lists only the jobs it can do, grouped by provider. Choose a job to see that list alone. An open job can use any registered model that fits it. The first model you allow closes that job to the models you tick. Workspaces use the same shape: a tick keeps the model, unticking it turns it off for that workspace. A name that follows whichever model is loaded says so. The SPEC-175 editor tests still pass, including the ceiling save and the workspace narrowing save.

A Vertex connection now saves its Google Cloud project (231)

The query page failed with GOOGLE_CLOUD_PROJECT is not set even when Vertex was saved, healthy, and offered. The embedding client was built from the process environment and never read the connection. A server-identity row stored nothing, so it was invisible to the fleet lookup.

Migration 231 adds provider_connections.google_project. The Connections form asks for the project, requires it for a server identity or an API key, and takes it from a service-account file when the field is left blank. Queries, chat, and gemini-embedding-2 use that saved project. GOOGLE_CLOUD_PROJECT is not read on that path. The deployment card shows Project {id}.

Row security (228–230)

An EXPLAIN-driven audit of the PostgreSQL query and index surface, run against a database built from the real migration chain and seeded to production-like scale: 3 tenants, 20 workspaces, 4k documents, 120k chunks, 2.2M BM25 postings, 120k halfvec(1536) embeddings with HNSW.

Three performance defects found, all fixed, measured, and guarded. A later commit realigns six pre-existing stale contract tests (no production source changed).

1. The clearance policy called a per-row function for a per-statement value (228)

Migration 220's restrictive clearance policy buried two row-independent terms inside the per-row clearance_covers(), so the planner evaluated them once per candidate row and no scalar subquery could be hoisted out.

Split at the point where row-independence ends: clearance_bypass() (row-independent), clearance_rank_covers() (per row), and clearance_covers() kept as bypass OR rank_covers so the three out-of-policy callers in graph/plain need no change. Policies call (SELECT clearance_bypass()), which the planner turns into an InitPlan β€” the technique migration 208 established.

2. The scope indexes did not cover the clearance columns (229)

The (tenant_id, workspace_id) indexes covered the scan but not the policy, so every candidate row was fetched from the heap only to be discarded β€” 18,000 heap blocks for one count. INCLUDE-ing the clearance columns makes it index-only. Each index replaces the existing one, so chunks stays at 16 indexes. Two byte-identical duplicate index pairs dropped.

Query (RLS-demoted, 36k chunks in one workspace) Before After
chunk count 291–407 ms 5–9 ms
chunk FTS page 263–270 ms 99–181 ms
document page 7.2–7.9 ms 0.8–1.8 ms
document count 6.6–6.7 ms 0.32–0.37 ms

3. Six policies still called session helpers bare (230)

Migration 208 wrapped session helpers in policy expressions and then documented an invariant: "New policies must use the wrapped form … e2e_rls_initplan_policies (edgequake-storage) fails if any are left bare." That guard test was never written, and five later migrations (215, 216, 219, 222, 225) reintroduced the bare form. Six policies were still re-invoking the helpers once per emitted row.

The plans show it directly: the index-only scan carried Filter: $0 over 4,031 rows on workspace_term_stats and 372,000 on chunk_terms. Wrapped, neither scan carries a filter and each helper is one InitPlan.

Query (RLS-demoted, median of 9 alternating runs) Before After
full BM25 statement 24.6 ms 12.4 ms (9/9 runs improved)

The composite statement gains what an isolated count(*) does not: a count reads only the index, while BM25 joins both tables and re-scans chunk_terms three times, so removing the per-row evaluation is where the time goes.

This PR also adds the guard test 208 promised, so the invariant cannot rot again.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective and that my feature works
  • New and existing tests pass locally with my changes

Additional context

Saved project, tested

  • Postgres: a_saved_project_embeds_without_google_cloud_project creates a fleet Vertex row with only google_project = mobioos-506015, loads it, and builds the gemini-embedding-2 client while GOOGLE_CLOUD_PROJECT is unset. The same test now builds the query embedder for a vertexai / gemini-embedding-2 workspace and checks the fleet client calls project mobioos-506015.
  • Postgres: the existing fleet create stores mobioos-506015 taken from the service-account file, and the secret update SQL still runs with the two new binds.
  • UI: the Connections form keeps Save disabled until a project is typed, and the POST body contains google_project. The saved row and the deployment card render Project {id}.
  • OpenAPI snapshot on this branch is 0.40.11 and includes google_project and base_url. The Edit form test saves the project with PUT and creates a connection for an environment provider with POST.
  • cargo clippy -p edgequake-api --features postgres --lib -- -D warnings is clean. Epoch coverage passes with the v0.40.10 tag (max=231, hash ba0c9db9d18d39c1, same migration set as v0.40.8). v0.40.11 adds no migration, so it serves that same set.

No regression, proven per change

  • Equivalence, exhaustively. 19,440 comparisons of the split clearance predicate against the verbatim migration-220 definition β€” 0 mismatches.
  • Suites from the audit. edgequake-storage --tests 180/180 Β· contract_spec150_train_head through 231 Β· e2e_tenant_rls Β· e2e_spec228_clearance_bypass Β· e2e_rls_initplan_policies.
  • Write cost unchanged. 2,000 chunk inserts measured 59–67 ms before and 59–66 ms after.

SPEC-001 Acc

Not re-scored. Mix retrieval ranking is unchanged. The latest valid pack remains medical-mid-20261024T031725Z (valid: true): EdgeQuake 0.8084, LightRAG 0.7681, delta +0.0403, 95% CI [-0.0241, +0.1036] (n_paired 72). The interval includes 0, so this is a statistical tie. EdgeQuake does not claim a win.

Checks that stay red

Security Audit (RUSTSEC-2026-0429, zip) and the E2E UI gate (Playwright has no Chromium build for ubuntu 26.04) are already red on edgequake-main. This release does not ignore the advisory. release-docker scans images with Trivy, not cargo audit.

Open in WebΒ Open in CursorΒ 

cursoragent and others added 8 commits November 7, 2026 04:31
Migration 220 put a restrictive clearance predicate on eight tables. Two
terms of that predicate are the same value for the whole statement (break
glass, and whether the workspace is enforcing), but they sat inside the
per-row clearance_covers() function, so the planner called the function for
every candidate row and a scalar subquery could not be hoisted out of it.

Split the predicate where row-independence ends: clearance_bypass() holds
the row-independent half, clearance_rank_covers() keeps rank and compartment
coverage, and clearance_covers() stays as bypass OR rank_covers so the
out-of-policy callers in graph/plain keep working unchanged. Policies call
the bypass as (SELECT clearance_bypass()), which the planner turns into an
InitPlan β€” the same technique migration 208 applied to current_tenant_id().

Migration 229 gives the (tenant_id, workspace_id) indexes on chunks and
documents the clearance columns as INCLUDE payload, so the policy is
answered from the index instead of a heap fetch per candidate row. Each
index replaces the existing one, so no table gains an index to maintain.
Two byte-identical duplicate index pairs are dropped.

Measured as edgequake_tenant_access via SET LOCAL ROLE (the path
rls::with_rls_transaction takes), 120k chunks / 36k in one workspace:

  chunk count           598 ms -> 6 ms
  chunk FTS page        267 ms -> 107 ms
  document page          13 ms -> 0.9 ms
  document count         12 ms -> 0.3 ms

Equivalence is proven exhaustively over every combination of enforcement,
break-glass, rank, allow-empty, compartments and workspace settings:
19,440 comparisons, 0 mismatches, against the verbatim migration-220
definition. e2e_spec228 additionally fails if the bypass is not hoisted.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
Adds a live-Postgres e2e test that compares the split predicate against
clearance_covers() across every combination of enforcement, break-glass,
rank, allow-empty, compartments and workspace settings, and asserts the
planner hoists the bypass (InitPlan `$n` before the per-row `SubPlan`) and
reaches the chunks scope index as an index-only scan.

Keeps the HEAD schema train contracts current: 229 is the head, the
release map covers 226 through 229, and two new contracts pin the shape of
migrations 228 and 229.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
Migration 208 wrapped session helpers in `(SELECT fn())` inside policy
expressions and documented the invariant: "New policies must use the
wrapped form ... e2e_rls_initplan_policies (edgequake-storage) fails if any
are left bare." That guard test was never written, and five later migrations
reintroduced the bare form β€” 215, 216, 219, 222 and 225 β€” so six policies
still call the helpers once per row.

A bare STABLE helper in a policy becomes a Filter the executor re-evaluates
for every emitted row. The plans show it: the index-only scan carries
`Filter: $0` over 4,031 rows on workspace_term_stats and 372,000 on
chunk_terms. Wrapped, both scans carry no filter and the helper is a single
InitPlan.

Measured as edgequake_tenant_access after SET LOCAL ROLE, 120k chunks and
2.2M chunk_terms rows, median of 9 alternating runs on an otherwise
identical pair of databases:

  full BM25 statement   24.6 ms -> 12.4 ms  (9 of 9 runs improved)

The composite statement gains what an isolated count(*) does not, because a
count reads only the index while BM25 joins both tables and re-scans
chunk_terms three times.

Meaning is unchanged: app.current_* are fixed for the statement, so the
wrapped call returns what the bare call returned. Verified on the three
states that could differ β€” unset, set and malformed GUC β€” and the rewrite
aborts if unwrapping does not reproduce the original expression.

Adds the guard test 208 promised, so this cannot rot again.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
e2e_rls_initplan_policies fails when any policy expression still calls a
session helper bare, naming the table, policy and clause. It fails on a
database before migration 230 and passes after β€” verified both directions.

Also pins 230 in the HEAD train contracts and the release map, and records
the fix in docs/migrations.md.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
Six contract tests asserted source-text literals that the code has since
outgrown. All six failed on edgequake-main before this branch existed, so
they were dead weight rather than a signal; each was confirmed unrelated to
the migrations here (no production source changed in this commit).

Each is realigned to the design now in place, keeping the invariant the test
existed to defend:

- incident edges: assert the OR template that replaced "OR ag_catalog"; the
  EDGE child table and no-UNION rules are unchanged.
- hot node reads: IMP-031 moved node reads off Cypher property MATCH onto
  native SQL (cypher_query_bound is now fallback-only); assert that.
- eq_* DDL early exit: assert the O(1) catalog probe short-circuits before
  any DDL, rather than one reworded log line.
- eq_* sync triggers: they moved to helpers::sync_triggers and use CREATE OR
  REPLACE, so assert no DROP TRIGGER on the hot path instead.
- discovery: migration 207 moved the GIN-eligible probe into a SECURITY
  DEFINER lookup; assert ?| plus the unnest/generate_series probe CTE there.
- vector DDL GUCs and PreferExisting: both were refactored into helpers and
  the provider; assert the behaviour at its new home.

edgequake-storage now passes 180/180 suites, up from 173 with 6 failing.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
Migration Guard / Layer 1 failed on this branch: check_epoch_coverage.sh
compares the [unreleased] pin against HEAD's numbered-migration set, and the
pin still described schema 227 / 225 files while HEAD is 230 / 228.

  unreleased pin 7491afd4e3a729e7 != HEAD db2868e7660eaf55 (max=230 files=228)

Repin to the measured HEAD hash. `./scripts/check_epoch_coverage.sh` now
reports "HEAD schema set: max=230 n=228 hash=db2868e7660eaf55 [OK unreleased]"
and every published tag still resolves to an epoch.

Nothing here is published-tag metadata: no [[epoch]] entry changes, so the
v0.40.x history keeps its own hash.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
Migration Guard / Layer 1's "Schema train docs parity" step failed: the
check compares the highest numbered migration (230) against manifest, then
requires the number to appear in docs/operations/upgrading.md,
edgequake/docs/migrations.md and edgequake/migrations/NOTES.md.

upgrading.md and NOTES.md still stopped at 227, and the per-migration table
in upgrading.md had no rows for 228-230. Adds the three rows and refreshes
both train headers. `./scripts/check_schema_train_docs.sh` passes.

Also states plainly that 228-230 are unreleased β€” no [[epoch]] entry is
added here, so the v0.40.x published-tag history keeps its own hash.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
A Vertex connection saves its Google Cloud project. Queries, chat, and
gemini-embedding-2 use that project, so a server without
GOOGLE_CLOUD_PROJECT can still answer. Schema moves 227 to 231
(migrations 228-231). Row-security InitPlans from 228-230 ship in this cut.

SPEC-001 Acc is not re-scored. Mix retrieval is unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursor cursor Bot changed the title perf(db): plan the clearance RLS policy once instead of per row release: v0.40.8 β€” saved Vertex project and clearance InitPlans Nov 7, 2026
Deployment Providers can edit a provider. Update saves the name, base URL,
key, and the Google Cloud project for Vertex. Schema stays 231.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursor cursor Bot changed the title release: v0.40.8 β€” saved Vertex project and clearance InitPlans release: v0.40.9 β€” edit a provider, saved Vertex project Nov 7, 2026
The v0.40.9 image publish was cancelled when quality-gates hit 90 minutes
on a cold compile. Raise that job limit to 180 minutes so the same
provider Edit can publish. Schema stays 231.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursor cursor Bot changed the title release: v0.40.9 β€” edit a provider, saved Vertex project release: v0.40.10 β€” edit a provider, saved Vertex project Nov 7, 2026
cursoragent and others added 2 commits November 7, 2026 12:46
The organization ceiling and the workspace list were a grid of empty
boxes. An empty box meant the job was open, which reads as off. Each
model now lists only the jobs it can do, grouped by provider, and a job
stays open until the first model is allowed.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
A Normal or image query built gemini-embedding-2 from the process
environment and ignored the project stored on the Vertex connection.
Query, chat, and image ingest now open that connection. Schema stays 231.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
@cursor cursor Bot changed the title release: v0.40.10 β€” edit a provider, saved Vertex project release: v0.40.11 β€” query uses the saved Vertex project Nov 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants