Repository navigation
release: v0.40.11 β query uses the saved Vertex project - #533
Open
raphaelmansuy wants to merge 12 commits into
Open
raphaelmansuy wants to merge 12 commits into
raphaelmansuy wants to merge 12 commits into
Conversation
Migration 220 put a restrictive clearance predicate on eight tables. Two terms of that predicate are the same value for the whole statement (break glass, and whether the workspace is enforcing), but they sat inside the per-row clearance_covers() function, so the planner called the function for every candidate row and a scalar subquery could not be hoisted out of it. Split the predicate where row-independence ends: clearance_bypass() holds the row-independent half, clearance_rank_covers() keeps rank and compartment coverage, and clearance_covers() stays as bypass OR rank_covers so the out-of-policy callers in graph/plain keep working unchanged. Policies call the bypass as (SELECT clearance_bypass()), which the planner turns into an InitPlan β the same technique migration 208 applied to current_tenant_id(). Migration 229 gives the (tenant_id, workspace_id) indexes on chunks and documents the clearance columns as INCLUDE payload, so the policy is answered from the index instead of a heap fetch per candidate row. Each index replaces the existing one, so no table gains an index to maintain. Two byte-identical duplicate index pairs are dropped. Measured as edgequake_tenant_access via SET LOCAL ROLE (the path rls::with_rls_transaction takes), 120k chunks / 36k in one workspace: chunk count 598 ms -> 6 ms chunk FTS page 267 ms -> 107 ms document page 13 ms -> 0.9 ms document count 12 ms -> 0.3 ms Equivalence is proven exhaustively over every combination of enforcement, break-glass, rank, allow-empty, compartments and workspace settings: 19,440 comparisons, 0 mismatches, against the verbatim migration-220 definition. e2e_spec228 additionally fails if the bypass is not hoisted. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
Adds a live-Postgres e2e test that compares the split predicate against clearance_covers() across every combination of enforcement, break-glass, rank, allow-empty, compartments and workspace settings, and asserts the planner hoists the bypass (InitPlan `$n` before the per-row `SubPlan`) and reaches the chunks scope index as an index-only scan. Keeps the HEAD schema train contracts current: 229 is the head, the release map covers 226 through 229, and two new contracts pin the shape of migrations 228 and 229. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
Migration 208 wrapped session helpers in `(SELECT fn())` inside policy expressions and documented the invariant: "New policies must use the wrapped form ... e2e_rls_initplan_policies (edgequake-storage) fails if any are left bare." That guard test was never written, and five later migrations reintroduced the bare form β 215, 216, 219, 222 and 225 β so six policies still call the helpers once per row. A bare STABLE helper in a policy becomes a Filter the executor re-evaluates for every emitted row. The plans show it: the index-only scan carries `Filter: $0` over 4,031 rows on workspace_term_stats and 372,000 on chunk_terms. Wrapped, both scans carry no filter and the helper is a single InitPlan. Measured as edgequake_tenant_access after SET LOCAL ROLE, 120k chunks and 2.2M chunk_terms rows, median of 9 alternating runs on an otherwise identical pair of databases: full BM25 statement 24.6 ms -> 12.4 ms (9 of 9 runs improved) The composite statement gains what an isolated count(*) does not, because a count reads only the index while BM25 joins both tables and re-scans chunk_terms three times. Meaning is unchanged: app.current_* are fixed for the statement, so the wrapped call returns what the bare call returned. Verified on the three states that could differ β unset, set and malformed GUC β and the rewrite aborts if unwrapping does not reproduce the original expression. Adds the guard test 208 promised, so this cannot rot again. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
e2e_rls_initplan_policies fails when any policy expression still calls a session helper bare, naming the table, policy and clause. It fails on a database before migration 230 and passes after β verified both directions. Also pins 230 in the HEAD train contracts and the release map, and records the fix in docs/migrations.md. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
Six contract tests asserted source-text literals that the code has since outgrown. All six failed on edgequake-main before this branch existed, so they were dead weight rather than a signal; each was confirmed unrelated to the migrations here (no production source changed in this commit). Each is realigned to the design now in place, keeping the invariant the test existed to defend: - incident edges: assert the OR template that replaced "OR ag_catalog"; the EDGE child table and no-UNION rules are unchanged. - hot node reads: IMP-031 moved node reads off Cypher property MATCH onto native SQL (cypher_query_bound is now fallback-only); assert that. - eq_* DDL early exit: assert the O(1) catalog probe short-circuits before any DDL, rather than one reworded log line. - eq_* sync triggers: they moved to helpers::sync_triggers and use CREATE OR REPLACE, so assert no DROP TRIGGER on the hot path instead. - discovery: migration 207 moved the GIN-eligible probe into a SECURITY DEFINER lookup; assert ?| plus the unnest/generate_series probe CTE there. - vector DDL GUCs and PreferExisting: both were refactored into helpers and the provider; assert the behaviour at its new home. edgequake-storage now passes 180/180 suites, up from 173 with 6 failing. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
Migration Guard / Layer 1 failed on this branch: check_epoch_coverage.sh compares the [unreleased] pin against HEAD's numbered-migration set, and the pin still described schema 227 / 225 files while HEAD is 230 / 228. unreleased pin 7491afd4e3a729e7 != HEAD db2868e7660eaf55 (max=230 files=228) Repin to the measured HEAD hash. `./scripts/check_epoch_coverage.sh` now reports "HEAD schema set: max=230 n=228 hash=db2868e7660eaf55 [OK unreleased]" and every published tag still resolves to an epoch. Nothing here is published-tag metadata: no [[epoch]] entry changes, so the v0.40.x history keeps its own hash. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
Migration Guard / Layer 1's "Schema train docs parity" step failed: the check compares the highest numbered migration (230) against manifest, then requires the number to appear in docs/operations/upgrading.md, edgequake/docs/migrations.md and edgequake/migrations/NOTES.md. upgrading.md and NOTES.md still stopped at 227, and the per-migration table in upgrading.md had no rows for 228-230. Adds the three rows and refreshes both train headers. `./scripts/check_schema_train_docs.sh` passes. Also states plainly that 228-230 are unreleased β no [[epoch]] entry is added here, so the v0.40.x published-tag history keeps its own hash. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
A Vertex connection saves its Google Cloud project. Queries, chat, and gemini-embedding-2 use that project, so a server without GOOGLE_CLOUD_PROJECT can still answer. Schema moves 227 to 231 (migrations 228-231). Row-security InitPlans from 228-230 ship in this cut. SPEC-001 Acc is not re-scored. Mix retrieval is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
Deployment Providers can edit a provider. Update saves the name, base URL, key, and the Google Cloud project for Vertex. Schema stays 231. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
The v0.40.9 image publish was cancelled when quality-gates hit 90 minutes on a cold compile. Raise that job limit to 180 minutes so the same provider Edit can publish. Schema stays 231. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
The organization ceiling and the workspace list were a grid of empty boxes. An empty box meant the job was open, which reads as off. Each model now lists only the jobs it can do, grouped by provider, and a job stays open until the first model is allowed. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
A Normal or image query built gemini-embedding-2 from the process environment and ignored the project stored on the Vertex connection. Query, chat, and image ingest now open that connection. Schema stays 231. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Raphael MANSUY <raphael.mansuy@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
v0.40.11 is tagged (
v0.40.11β926d357b6). Schema stays 231. Upgrade notes: docs/operations/upgrade-to-0.40.11.md.A query on a Normal or image workspace still failed with
GOOGLE_CLOUD_PROJECT is not setafter the project was saved on the Vertex card. The query builtgemini-embedding-2from the process environment and never opened the connection. Query, chat, and image ingest now use the project saved on that connection. A stored service account is used when one is saved. The image publish is Release β Docker (GHCR). Deploy GCP rolls the demo after that workflow succeeds, because the tag branch isv0.40.11.The organization model ceiling and the workspace model list, arranged by job, ship in this same image.
v0.40.10 is tagged (
v0.40.10β3812ed443). Schema stays 231. Upgrade notes: docs/operations/upgrade-to-0.40.10.md.The v0.40.9 image publish (Release β Docker) was cancelled when quality-gates hit its 90 minute limit during a cold compile. The tests had passed. v0.40.10 raises that limit to 180 minutes. The new publish is Release β Docker (GHCR). Deploy GCP rolls the demo after that workflow succeeds.
Edit a provider
Deployment βΊ Providers had Refresh and Remove, and no way to change a provider. Each card now has Edit. Update saves the name, base URL, a replacement key, and, for Vertex, the Google Cloud project. Leave the key blank to keep the key the server already has. The demo picks this up when the v0.40.10 image publish finishes and Deploy GCP rolls it.
v0.40.8 is tagged (
v0.40.8βba38afcb2). Schema moves 227 β 231. Runedgequake migratebefore the new API replicas. Upgrade notes: docs/operations/upgrade-to-0.40.8.md.The image publish for v0.40.8 is Release β Docker (GHCR). v0.40.9 was cancelled at the 90 minute gate (run). v0.40.10 is Release β Docker (GHCR).
Model ceiling, by job
The organization ceiling was a grid of empty boxes. An empty box meant the job was open, so a long list of chat models looked like nothing was allowed, and a dash hid the difference between "cannot do this job" and "not chosen."
Each model now lists only the jobs it can do, grouped by provider. Choose a job to see that list alone. An open job can use any registered model that fits it. The first model you allow closes that job to the models you tick. Workspaces use the same shape: a tick keeps the model, unticking it turns it off for that workspace. A name that follows whichever model is loaded says so. The SPEC-175 editor tests still pass, including the ceiling save and the workspace narrowing save.
A Vertex connection now saves its Google Cloud project (231)
The query page failed with
GOOGLE_CLOUD_PROJECT is not seteven when Vertex was saved, healthy, and offered. The embedding client was built from the process environment and never read the connection. A server-identity row stored nothing, so it was invisible to the fleet lookup.Migration 231 adds
provider_connections.google_project. The Connections form asks for the project, requires it for a server identity or an API key, and takes it from a service-account file when the field is left blank. Queries, chat, andgemini-embedding-2use that saved project.GOOGLE_CLOUD_PROJECTis not read on that path. The deployment card showsProject {id}.Row security (228β230)
An EXPLAIN-driven audit of the PostgreSQL query and index surface, run against a database built from the real migration chain and seeded to production-like scale: 3 tenants, 20 workspaces, 4k documents, 120k chunks, 2.2M BM25 postings, 120k
halfvec(1536)embeddings with HNSW.Three performance defects found, all fixed, measured, and guarded. A later commit realigns six pre-existing stale contract tests (no production source changed).
1. The clearance policy called a per-row function for a per-statement value (228)
Migration 220's restrictive clearance policy buried two row-independent terms inside the per-row
clearance_covers(), so the planner evaluated them once per candidate row and no scalar subquery could be hoisted out.Split at the point where row-independence ends:
clearance_bypass()(row-independent),clearance_rank_covers()(per row), andclearance_covers()kept asbypass OR rank_coversso the three out-of-policy callers ingraph/plainneed no change. Policies call(SELECT clearance_bypass()), which the planner turns into an InitPlan β the technique migration 208 established.2. The scope indexes did not cover the clearance columns (229)
The
(tenant_id, workspace_id)indexes covered the scan but not the policy, so every candidate row was fetched from the heap only to be discarded β 18,000 heap blocks for one count.INCLUDE-ing the clearance columns makes it index-only. Each index replaces the existing one, so chunks stays at 16 indexes. Two byte-identical duplicate index pairs dropped.3. Six policies still called session helpers bare (230)
Migration 208 wrapped session helpers in policy expressions and then documented an invariant: "New policies must use the wrapped form β¦ e2e_rls_initplan_policies (edgequake-storage) fails if any are left bare." That guard test was never written, and five later migrations (215, 216, 219, 222, 225) reintroduced the bare form. Six policies were still re-invoking the helpers once per emitted row.
The plans show it directly: the index-only scan carried
Filter: $0over 4,031 rows onworkspace_term_statsand 372,000 onchunk_terms. Wrapped, neither scan carries a filter and each helper is one InitPlan.The composite statement gains what an isolated
count(*)does not: a count reads only the index, while BM25 joins both tables and re-scanschunk_termsthree times, so removing the per-row evaluation is where the time goes.This PR also adds the guard test 208 promised, so the invariant cannot rot again.
Type of change
Checklist
Additional context
Saved project, tested
a_saved_project_embeds_without_google_cloud_projectcreates a fleet Vertex row with onlygoogle_project = mobioos-506015, loads it, and builds thegemini-embedding-2client whileGOOGLE_CLOUD_PROJECTis unset. The same test now builds the query embedder for avertexai/gemini-embedding-2workspace and checks the fleet client calls projectmobioos-506015.mobioos-506015taken from the service-account file, and the secret update SQL still runs with the two new binds.google_project. The saved row and the deployment card renderProject {id}.google_projectandbase_url. The Edit form test saves the project with PUT and creates a connection for an environment provider with POST.cargo clippy -p edgequake-api --features postgres --lib -- -D warningsis clean. Epoch coverage passes with the v0.40.10 tag (max=231, hashba0c9db9d18d39c1, same migration set as v0.40.8). v0.40.11 adds no migration, so it serves that same set.No regression, proven per change
edgequake-storage --tests180/180 Β·contract_spec150_train_headthrough 231 Β·e2e_tenant_rlsΒ·e2e_spec228_clearance_bypassΒ·e2e_rls_initplan_policies.SPEC-001 Acc
Not re-scored. Mix retrieval ranking is unchanged. The latest valid pack remains
medical-mid-20261024T031725Z(valid: true): EdgeQuake 0.8084, LightRAG 0.7681, delta +0.0403, 95% CI [-0.0241, +0.1036] (n_paired 72). The interval includes 0, so this is a statistical tie. EdgeQuake does not claim a win.Checks that stay red
Security Audit (
RUSTSEC-2026-0429,zip) and the E2E UI gate (Playwright has no Chromium build for ubuntu 26.04) are already red onedgequake-main. This release does not ignore the advisory.release-dockerscans images with Trivy, notcargo audit.