[AbirAbbas]

Carries #1178 (merged into the prototype branch proto/harness-watch on 2026-11-10) to main; it is the one commit of that branch that #1174 did not bring over. Cherry-picked unchanged, with its author kept.

Why

When a node whose work survives process stops or machine sleep (such as a local factory node on a laptop that sleeps overnight) goes offline, nothing renews its lease while it is away. Previously, the control plane's stale-execution sweep timed out those running executions once stale_execution_timeout elapsed, even though the node still held the work on disk and would resume reporting on it as soon as it reconnected.

What Changes

• Per-node opt-in via registration metadata: A node may register with metadata.custom.holds_executions_offline: true (types.AgentMetadataHoldsExecutionsOffline, checked by (*AgentNode).HoldsExecutionsOffline()).
• Holding executions while an opted-in node is offline: Before each stale cleanup pass runs, ExecutionCleanupService checks registered nodes. If an opted-in node's heartbeat/lease has lapsed (last_seen older than the lease TTL), its quiet running/pending/queued executions (on both executions and workflow_executions) are marked held (status stays running, status_reason is set to node_offline, and updated_at records when the hold began).
• Stale sweeps skip held rows: MarkStaleExecutions, MarkStaleWorkflowExecutions, and RetryStaleWorkflowExecutions exclude rows with status_reason = 'node_offline'.
• Four ways a hold ends:
  1. Node returns with the execution: On the node's next lease (PATCH /api/v1/nodes/:node_id/status, including link hello/heartbeat), any held execution listed in inflight is released (status_reason cleared to NULL, updated_at refreshed) and continues running.
  2. Node returns without the execution: Any held execution omitted from the node's inflight list fails immediately with status_reason = "node_lost_execution".
  3. Node is unregistered: If the node is no longer registered on a subsequent cleanup pass, its held executions fail with status_reason = "node_removed".
  4. Hold exceeds the ceiling: If the node stays offline longer than offline_hold_ceiling (default 168h / 7 days, configurable via agentfield.execution_cleanup.offline_hold_ceiling in agentfield.yaml or AGENTFIELD_EXECUTION_OFFLINE_HOLD_CEILING), its held executions fail with status_reason = "node_offline_too_long".

Who Is Unaffected and Why

• All existing agent deployments and older SDKs (including SWE-AF, pr-af, sec-af, and standard Python/Go/TypeScript SDK nodes) are completely unaffected.
• The behavior is strictly opt-in via metadata.custom.holds_executions_offline: true. Nodes that do not set this flag return false from HoldsExecutionsOffline(), never have their executions marked node_offline, and have their quiet executions reaped by the stale sweeps (MarkStaleExecutions, MarkStaleWorkflowExecutions, RetryStaleWorkflowExecutions) identically to before.
• Even for an opted-in node, if its lease is current (the node is online) and it stops listing an execution in inflight, the execution is not held and is timed out by the stale sweep as before.
• Existing control planes upgraded in place require no config changes (offline_hold_ceiling defaults to 168h when unset in YAML or database config) and no database schema migration.

Rollback

Revert the commit. There is no schema migration or new table/column to undo; any execution currently marked status_reason = 'node_offline' remains a standard running row in executions and workflow_executions and will fall back under the ordinary stale-execution sweep on the next cleanup pass.

What Was Run

• go build ./... (with GOFLAGS=-buildvcs=false) in control-plane
• go vet ./... in control-plane
• golangci-lint run on the touched packages (./internal/handlers, ./internal/storage, ./internal/config, ./internal/server, ./pkg/types)
• go test -tags sqlite_fts5 ./internal/handlers ./internal/storage ./internal/config ./internal/server ./pkg/types in control-plane

🤖 Generated with Claude Code

[github-actions]

📊 Coverage gate

Thresholds from .coverage-gate.toml: per-surface ≥ 84%, aggregate ≥ 85%, max per-surface regression ≤ 1.0 pp, max aggregate regression ≤ 0.50 pp.

Surface	Current	Baseline	Δ
control-plane	88.20%	87.40%	↑ +0.80 pp	🟡
sdk-go	93.30%	92.00%	↑ +1.30 pp	🟢
sdk-python	94.77%	93.73%	↑ +1.04 pp	🟢
sdk-typescript	92.44%	90.42%	↑ +2.02 pp	🟢
web-ui	84.90%	84.79%	↑ +0.11 pp	🟡
aggregate	86.12%	85.75%	↑ +0.37 pp	🟡

✅ Gate passed

No surface regressed past the allowed threshold and the aggregate stayed above the floor.

[github-actions]

📐 Patch coverage gate

Threshold: 80% on lines this PR touches vs origin/main (from .coverage-gate.toml:thresholds.min_patch).

Surface	Touched lines	Patch coverage	Status
control-plane	297	95.00%	✅
sdk-go	0	—	➖ no changes
sdk-python	0	—	➖ no changes
sdk-typescript	0	—	➖ no changes
web-ui	0	—	➖ no changes

✅ Patch gate passed

Every surface whose lines were touched by this PR has patch coverage at or above the threshold.

[santoshkumarradha]

Thanks for carrying this behavior through the cleanup and lease paths. The required coverage-summary check is currently failing because patch coverage is 58%, below the repository’s 80% floor. Please add focused tests for the uncovered offline-hold branches, then rerun the coverage gate. Once that is green, this should be ready for another look.