Repository navigation
perf(control-plane): fast Runs list and run page under live traffic - #1172
Closed
santoshkumarradha wants to merge 4 commits into
Closed
santoshkumarradha wants to merge 4 commits into
santoshkumarradha wants to merge 4 commits into
Conversation
Every execution event invalidated the runs list, the dashboard summary, the run DAG and the step detail at once, with cancelRefetch on, so under steady traffic the Runs page restarted its list request dozens of times a minute and each cancelled request still ran to completion on the server. Invalidations are now coalesced (at most one flush per two seconds, the first immediate, without cancelling an in-flight fetch), and a running run polls every ten seconds as a safety net while the event stream is connected.
QueryRunSummaries grouped every execution in the table, computing every summary column for every run, and then kept one page. It now chooses the page's run_ids first, computing only the sort key, and aggregates just those runs with the same filters. The default page (no filters, latest activity first) is read from a new expression index on COALESCE(updated_at, started_at) and stops as soon as the page is complete. run_id breaks sort ties so page boundaries no longer move between requests. Tests compare every sort column, direction, filter and page against the previous single grouped query.
SQLite compares started_at as the text the driver wrote, and the driver writes a time in the zone it carries. An execution whose start time was not UTC was filtered and sorted as though it started at a different instant; a since bound given in UTC dropped it. Insert and update now write the UTC instant. Rows already stored are unchanged.
The run page fetched the whole workflow VC chain, one signed credential per step, to show its verifiability warning and issuer DID: about 1.2 MB for a 1500-step run, on every page open. GET /api/ui/v1/workflows/:id/vc-chain accepts view=summary, which returns the same status, step counts, verifiability and workflow issuer without component_vcs, the workflow credential's document or the DID resolution bundle. The default response is unchanged, and the audit download still fetches the full chain.
19 of 22 tasks
Member
Author
|
Folded into #1171 so all fixes get one review; commits unchanged. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| } | ||
| defer rows.Close() | ||
|
|
||
| runIDs := make([]string, 0, limit) |
Contributor
📊 Coverage gateThresholds from
✅ Gate passedNo surface regressed past the allowed threshold and the aggregate stayed above the floor. |
Contributor
📐 Patch coverage gateThreshold: 80% on lines this PR touches vs
✅ Patch gate passedEvery surface whose lines were touched by this PR has patch coverage at or above the threshold. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Under steady traffic the Runs page and a large run's page got slow. Three things caused it: the runs list aggregated every execution in the table on each request, every live event restarted the page's queries, and the run page downloaded the whole VC chain to show one warning.
Notes:
run_id. Before, tie order was undefined, so a run could show up on two pages or on neither. This matters most forsort_by=status, which has only three key values.Type of change
Evidence
Measured on a copy of a busy local SQLite database: 4.2 GB, about 24,000 runs and 203,000 executions. Live traffic came from a real agent through
/api/v1/execute/async: one 1,500-step run (six steps in parallel) plus a new 5-step run about every 0.66 s. The machine's load average was about 20 from unrelated work, so these are noisy-box numbers.GET /api/ui/v2/workflow-runs?page=1&page_size=25&sort_by=updated_at&sort_order=descQueryRunSummariesalone, default pageCOUNT(DISTINCT run_id)total)QueryRunSummaries, other sort columnsThe per-run lookups in the handler (run metadata and trigger) were measured at about 1–3 ms for a page of 25, so they were left as they are.
Network trace, 15 s on each page under traffic (headless Chrome,
/apirequests only)Runs page:
Run page of the 1,500-step run while it runs:
For a 322-step run, the full chain was 766 KB.
Test plan
go build ./...go vet ./internal/storage ./internal/handlers/uigo test ./internal/storage/ ./internal/handlers/ui/ ./internal/server/ -count=1POSTGRES_TEST_URL=... go test ./internal/storage -run Postgres -count=1againstpgvector/pgvector:pg17, including the new PostgreSQL page comparisonTestQueryRunSummariesPagesMatchTheSingleGroupedQuerycompares every sort column, both directions, ten filters and five page windows against the previous single grouped query. It fails if either tie-breaker is reversed.TestRecentRunIDsByActivityWidensPastBusyRuns,TestRunActivityIndexServesTheRecentRunsScanTestExecutionStartedAtIsStoredInUTCfails onmain(a non-UTC start drops out of a UTCsincefilter) and passes hereTestUIWorkflowVCChainSummaryViewOmitsPerStepCredentialsweb/client:npx tsc --noEmit -p tsconfig.app.json,npx vitest run(141 files, 840 tests),npm run buildMerge Danger
Door: two-way. Every change reverts cleanly. The new index is additive (
CREATE INDEX IF NOT EXISTS) and can be dropped.view=summaryis opt-in.Blast Radius: moderate. This changes the query behind the Runs list and adds one index on
executions. On a large PostgreSQL table, the first start after upgrading builds that index synchronously, the same way the existing startup indexes are built.