Repository navigation
WorkAF harness watch: node link, relay pipe and steer, streamed only for owned runs - #1174
Merged
Merged
Conversation
Slice 0: the trigger dispatcher sends Authorization: Bearer <internal_token>
like the execute controller and cancel dispatcher; the lease PATCH accepts
inflight and touches those executions' updated_at (owner and non-terminal
only), so the stale reaper leaves live work alone.
Slice 1 (prototype): GET /api/v1/nodes/:node_id/link, a node-opened
WebSocket authenticated by the API key or the internal token (header only).
Frames deliver{dispatch,cancel}, ack, status, progress, log, heartbeat.
The trigger and cancel dispatchers send down a live link and fall back to
HTTP base_url otherwise. Deliveries wait in an in-memory outbox and are
redelivered until acked; acks settle once. Up-frames re-enter the router as
the HTTP calls they replace.
SDK: codeaf harness Live hooks (--events/--steer tail, verbatim lines), agent steer mailbox + /_internal/executions/:id/steer, node link client (accepts steer, cancel). CP (half-done, handed to the cp lane): link as HTTP carrier for <node>.nodelink hosts, relay table (store-and-forward, seq/ack), steer route. Two SDK agent tests still red (see HANDOFF-cp.md).
…dupe, link-only mode, watched runs on chat --once A codeaf harness run's conversation-record lines are relayed verbatim to POST /executions/:id/relay, numbered from 1 by the node and replayed until answered (no loss, no doubles across a control plane outage). The link serves kind http on the agent's own handler; a redelivered id is answered with the same ack and never run twice. AGENTFIELD_NODE_LINK_ONLY=1 binds loopback and registers http://<id>.nodelink. A watched run uses codeaf chat --once (the native step engine) and falls back to exec on an older codeaf.
…e kit workaf-support: a generic_bearer ticket trigger runs triage (plain code), investigate (codeaf harness, calls the nested search_docs reasoner) and draft_reply (codeaf harness: fix, test, customer reply). workaf-research: plan, three parallel investigate_* harness calls, synthesize. Both are plain SDK use. deploy/workaf-proto: build, control plane on spark, node on dumb, codeaf engine on mac-studio; idempotent start/stop/clean, mode-600 secrets, listener proofs.
…nk-only nodes Link-only nodes: a node with no base_url is reached down its link for execute (sync and async), nested calls, trigger dispatch and steer. agenturl.ForNode maps an empty base to the node's link host when a carrier is installed; the carrier sends an http delivery to a node that accepts it and a dispatch delivery (ack read back as 202/200/422) to one that speaks only dispatch and cancel; a live link passes the node-is-down gate. An http delivery is resent on a new link only, not on a timer. The pipe (nodelink/pipe.go): entries up from the node (entries frame or POST /executions/:id/relay), idempotent on (execution, seq), held in execution_relay only until the run's owner acks them, ceilinged by the execution-log retention. Owners are per run (run_owners: the X-AgentField-Run-Owner on an execute, or a trigger's target node), so every descendant is in scope and nothing of another run reaches a subscriber. subscribe / entries_ack / steer up-frames; entries and run (tree, status, timings, result) down-frames pushed as they happen. Steer is answered only once the node took it, and refused in person words when it cannot. Tests (-race): execute over the link, seq dedupe, delete after ack, retention ceiling, resume after reconnect, a restart losing and doubling nothing, subscription scoping, steer ack and refusal, trigger dispatch to a link-only SDK node.
The research run's coding agents ran brew install zig and docker pulls on the node box and filled its shared Go caches. The node now gets its own GOPATH, GOCACHE and GOMODCACHE under the kit directory, the probe prompt forbids system-wide installs, and clean removes the tunnel's known_hosts row without ssh-keygen (which refuses a file with any invalid line).
The trigger dispatcher names the workflow but no run id, so every child started a run of its own with no parent and the run's tree was one node. A call with no run id now uses the workflow id before minting one.
A chat --once that names no session resumes the newest conversation in its directory, and a node reuses its clone of a repository from run to run, so the second ticket's step opened on the first ticket's whole turn. Each watched run now names --session in its own run directory, not yet created, so codeaf starts it fresh.
… cannot kill the engine The node runs link-only and prove checks every listener. The engine driver connects GitHub on the demo repository, turns the runner on and adds a generic_bearer trigger aimed at the factory's intake; fire-workaf.sh posts a ticket to the trigger this engine added. rpc.py makes one call through the driver's ctl fifo, naming its reply by absolute path; a bad ctl line is logged and never takes the engine down. The node's codeaf home sets every model row so no helper tier falls back to a built-in default.
…lback Each batch of a codeaf run's record is an entries frame on the link the node already holds: its data seqs in the body, the frame's seq the node's own counter that the control plane's ok answers. The node forgets what the ok's through covers. With no link up, or no answer on it, the same entries go by POST /executions/:id/relay. A 400 (an entry the control plane can never take) is logged and let go instead of holding every later line.
down.sh left workaf-research running on dumb once; stopping by the binary's path catches a node whose pid file no longer names it.
The dispatcher filed the root under run_id = workflow id but sent no X-Run-ID, so the SDK minted a fresh run for every child it called and the CP could not link any child to its parent: a trigger run's DAG had one node. Both the HTTP and the link dispatch now send X-Run-ID = the workflow id.
DiagnoseOutputFailure printed the expected and actual top-level key sets, which are equal apart from map order whenever the keys are right and a value is wrong, so the model retried blind; two research steps failed on it. The runner now reports the decode error into the result type and the JSON Schema violation, and the key fallback lists only missing and unexpected keys.
The desk shows the node id, and workaf-support read like machinery. The SDK registers the machine it runs on (AGENTFIELD_NODE_HOST, else the short hostname) as metadata.custom.host, so a run reads as support · dumb. The kit README no longer describes the ssh tunnel: the node is link only.
StructToJSONSchema said only array for a slice and object for a map or a nested struct, so the live support run's model wrote objects where Findings wanted strings in files, failed to decode, and spent a schema retry. The schema now carries items, additionalProperties and nested properties, with time.Time as a date-time string and a type that holds itself ending in a plain object.
The research example explains each Measurement field in a description tag that the model never saw.
A node drains its running harness on SIGTERM and outlived the check that followed; stop now waits up to ten seconds and then kills it.
The question is the call's question, else a WorkAF job's title and body; the SQLite default is only for a call with neither. The plan names the two to four options the question asks about, and one investigate reasoner runs per option at the same time, so a JSON-library ticket investigates JSON libraries. fire-workaf.sh takes a ticket title as its fourth argument.
The approval callback to a .nodelink host now goes down the node's link (agenturl's carrier) instead of the SSRF-safe dialer, which could never reach it; every callback retries with backoff for about five minutes and a final failure is stored as execution.approval_callback_failed. request-approval carries the question: kind (plan_review, question, approval), question, a markdown body and option labels, kept in a lazily created execution_approval_questions table and returned by approval-status with the request id and the answerer's feedback. approval_request_id is optional; the control plane mints apr_<hex>. request_changes keeps the round's id so the status still reads it. Waiting and resolved approvals go on the global execution bus, so the link pipe pushes a run frame whose ExecState carries status_reason and the pending approval. approval-response holds a caller that names itself with X-AgentField-Run-Owner to the runs it owns, as steer does.
PauseOptions gains Kind, Question, Body and Options, sent on request-approval; ApprovalRequestID is optional and the control plane's minted id is used. Pause also reads approval-status every PollInterval (5s by default) and resumes from it when the callback never reaches the node, with the answerer's feedback. ApprovalStatusResponse gains the question fields, the round's id and feedback; Response is raw JSON (as a map it could not decode the control plane's string).
app.pause and client.request_approval take kind, question, body and options; approval_request_id is optional and the control plane's minted id is used. get_approval_status returns the question, the round's id and the answerer's feedback, and wait_for_resume's poll carries that feedback and ignores a status for another round.
Cancelling an execution wrote status_reason only when the caller gave a reason. A person's Stop gives none, so the field kept what the run said last: a WorkAF run stopped after its plan was approved reported root_error_category "approval_granted" (live run, 2026-11-07). Both the single cancel and the tree cancel now always write a reason: the caller's, or "cancelled" (types.ExecutionReasonCancelled). The response and the cancelled event still carry only the caller's own reason.
Agent.Harness armed the live path for every call made inside an execution, so every agent using the codeaf harness switched from `codeaf exec` to `codeaf chat --once --yolo --no-host` and relayed its transcript to the control plane with nobody reading it. The control plane now marks an execution whose run has an owner with X-AgentField-Run-Watched: 1. The SDK reads it into ExecutionContext.RunWatched (carried to local child contexts), and liveHarness arms only when it is set. An unwatched run makes the same command line as before the live path existed, sends no relay requests, and refuses a steer with ErrNotSteerable. An older control plane never sends the header, so nothing streams. gorilla/websocket moves to a direct requirement: the node link uses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay pipe stored harness transcript lines for executions whose run had no owner, where they sat until the 24h log retention with nobody to ack them, and it pruned the relay table on every write. Every carrier that delivers an execution to a node (execute sync and async, restarts, resumed runs, the legacy /reasoners and /skills routes, trigger dispatch over HTTP and over the link) now sets X-AgentField-Run-Watched: 1 exactly when the run has a recorded owner, with positive owner lookups cached because an owner never moves. X-AgentField-Run-Owner is never forwarded to a node, since on requests to the control plane it claims or scopes ownership. Pipe.Put stores nothing for a run with no owner and answers through = the batch's highest seq, so the node forgets those lines instead of retrying; a store error is still returned. RelayPrune runs at most once a minute per pipe. The legacy /reasoners and /skills routes record X-AgentField-Run-Owner on a root call, as /execute already did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Performance
✓ No regressions detected |
wait_for_resume's poll fallback now reads the round's approval_request_id and the answerer's feedback off ApprovalStatusResponse, which always carries both. The test stood the response up as a SimpleNamespace with only status and response, so it failed with an AttributeError no control plane can produce. It now builds the real dataclass, naming the round it asked about. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
📊 Coverage gateThresholds from
✅ Gate passedNo surface regressed past the allowed threshold and the aggregate stayed above the floor. |
Contributor
📐 Patch coverage gateThreshold: 80% on lines this PR touches vs
✅ Patch gate passedEvery surface whose lines were touched by this PR has patch coverage at or above the threshold. |
The harness-watch storage and routes landed without tests, so the PR's patch coverage for control-plane sat at 68% against the 80% floor. New tests, each written from a stated behaviour: relay batches read back in seq order, a repeated seq kept once, an ack dropping only that execution's rows, prune by age, counts and the executions listing; a run's first owner kept and an unknown run reading empty; RunsOwnedBy scoped by owner and time; approval questions kept per round with the newer words on a re-ask; both tables created lazily; the HTTP relay, ack, count and steer routes held to the run's owner; lookupRunWatched caching only positive answers; a cancel sent down a node's live link; and the API-key middleware letting only the exact /nodes/:id/link path through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AbirAbbas
added a commit
that referenced
this pull request
Nov 10, 2026
…ever cache a failed create (#1177) The three tables #1174 added (execution_relay, run_owners, execution_approval_questions) were created lazily with CREATE TABLE IF NOT EXISTS. On Postgres two first uses raced on pg_type_typname_nsp_index, and the store cached the error, so run ownership and relay writes stayed broken until a restart; ordinary executions also ran DDL on every call. The tables are now created at startup in autoMigrateSchema for SQLite and Postgres. The remaining guard serializes creation per db, caches only success, runs no DDL in steady state, and treats a concurrent create by another process as success. Proved by the existing-deployments smoke suite (15/15 on main plus this change). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the WorkAF harness-watch work (originally
proto/harness-watch) withmainmerged in, and makes the live harness path opt-in: a codeaf harness run is streamed only when its run is owned by a subscriber, which today is WorkAF'scodeaf-factorynode. Every other agent keeps running codeaf exactly as it does onmain.What this brings
control-plane/internal/nodelink,sdk/go/agent/link.go): a node opens one long-lived WebSocket to the control plane, so a node behind NAT, or one with no inbound port at all (AGENTFIELD_NODE_LINK_ONLY=1), still receives dispatches, nested calls, cancels and steers. Redeliveries are idempotent.POST /api/v1/executions/:id/relay, or an entries frame on the link). The control plane holds them only until the run's owner acks them; the owner is the system of record. A person's steering note reaches the running codeaf session throughPOST /api/v1/executions/:id/steer.Pausecan send kind, question, markdown body and option labels to the control plane, andapproval_request_idis optional, so a run with no external approval service can still pause and be answered from any door.X-Run-ID, reaches a linked node down its link, and a trigger-started run's children join its run tree.examples/workaf-support,examples/workaf-research,deploy/workaf-proto/.The opt-in fix (last two commits)
On the prototype,
Agent.Harnessarmed the live path for every call made inside an execution. Every agent using the codeaf harness switched fromcodeaf exectocodeaf chat --once --yolo --no-hostand streamed its transcript with nobody reading it. The relay also stored lines for runs with no owner until the 24h log retention, and pruned the table on every write.X-AgentField-Run-Watched: 1, on every request that delivers an execution to a node, but only when the execution's run has a recorded owner (run_owners). The carriers covered:/executesync and async, restarts, resumed runs, the legacy/reasonersand/skillsroutes, trigger dispatch over HTTP and over the link.X-AgentField-Run-Owneritself is never forwarded to nodes.ExecutionContext.RunWatched, andliveHarnessarms only when it is set. An unwatched run makes themaincommand line and zero relay requests, and a steer to it is refused withErrNotSteerable.Pipe.Putstores nothing for a run with no owner and answersthrough= the batch's highest seq, so the node forgets the lines instead of retrying. A store error is still returned, so a real failure is retried.RelayPruneruns at most once a minute per pipe./reasonersand/skillsroutes now recordX-AgentField-Run-Owneron a root call, as/executealready did, so a run started through them can be owned too.test_wait_for_resume_falls_back_to_status_pollbuilds the realApprovalStatusResponseinstead of aSimpleNamespace: the fallback now reads the round id and feedback, which the real type always carries.Compatibility
main.execution_relay,run_ownersand the approval-question table are created lazily; existing migrations are untouched./api/v1/nodes/:id/linkbypasses the API-key middleware and authenticates its own upgrade, with the API key or the internal token.pause():approval_request_idbecame optional and the new fields are keyword arguments, so existing calls are unaffected.Open before merge
deploy/workaf-proto/and the twoexamples/workaf-*nodes should land inmainor move out; they are prototype kits.Type of change
Test plan
cd sdk/go && go test -count=1 -run '^(TestHarnessOnCodeAFStreamsItsTranscriptAndTakesASteer|TestHarnessOnAnotherProviderIsNeverSteerable|TestAgentStreamsOnlyWatchedRuns|TestHarnessOutsideExecutionContextLeavesLiveNil)$' ./agent/ ./harness/cd control-plane && go test -tags sqlite_fts5 -count=1 -run '^(TestPutDropsUnownedEntriesWithoutRetryAndHoldsOwnedEntries|TestRelayPruneRunsAtMostOncePerMinute|TestRetentionIsTheCeiling|TestExecuteReachesALinkOnlyNodeOverTheLink|TestExecuteSetsRunWatchedOnlyWhenRunHasOwner|TestTriggerDispatchSetsRunWatchedOnlyWhenRunHasOwner)$' ./internal/nodelink/ ./internal/handlers/ ./internal/services/cd sdk/python && pytest tests/test_agent_coverage_additions.py tests/test_approval.py tests/test_client_laser_push.py(49 passed)go build ./...forsdk/go;go build ./internal/... ./pkg/...forcontrol-plane;gofmt -lclean on changed filesbuild_issue, once SWE-AF builds against this SDK🤖 Generated with Claude Code