[santoshkumarradha]

Why

A node that is offline for a while (a laptop asleep overnight, an engine quit for hours) currently gets its running executions failed by the stale sweep. Nothing renews them while the node is away, so ExecutionCleanupService.performCleanup (internal/handlers/execution_cleanup.go:179 and :189) calls MarkStaleExecutions / MarkStaleWorkflowExecutions. Those select every running|pending|queued row whose updated_at is older than stale_execution_timeout (internal/storage/execution_records.go:1274 and :1429) and set them to timeout with "execution timed out (no activity)". The only thing that keeps a row alive is the node's own lease listing it as inflight, and a sleeping node sends no lease. Agent-Field/CodeAF#2573 made the engine keep its instance id across restarts, so the restart reap no longer fires. This sweep was the remaining gap.

What changes

The fix is opt-in per node. A node that registers with metadata.custom.holds_executions_offline: true promises that its work survives being offline, and that on its first lease back it lists every execution it still holds as inflight. Nodes that do not opt in behave exactly as before.

• Hold. Each cleanup pass, before the retry and stale sweeps, checks every opted-in, registered node whose last heartbeat is older than the lease TTL (5m). Its quiet executions are marked held: status stays running, status_reason = node_offline, and updated_at records when the hold began (the UI can show "node offline since"). This is written to executions and workflow_executions together.
• The sweeps skip held rows. MarkStaleExecutions, MarkStaleWorkflowExecutions and RetryStaleWorkflowExecutions each exclude status_reason = 'node_offline'.
• A hold ends one of four ways:
  • The node's lease lists the execution as inflight: it is released, the reason is cleared, and it carries on.
  • The lease does not list it: it is failed with node_lost_execution ("node X came back without this execution: it was lost while the node was offline").
  • The node is no longer registered: it is failed with node_removed.
  • The hold outlives execution_cleanup.offline_hold_ceiling (default 168h, env AGENTFIELD_EXECUTION_OFFLINE_HOLD_CEILING): it is failed with node_offline_too_long.
• An opted-in node that is online but stops listing an execution is still reaped by the normal stale sweep. "Offline" is the condition for a hold, not the opt-in alone.
• The lease response now also returns held_released and held_lost.

Notes

• A row that was already held through an agent restart (awaiting_agent_restart) is never re-held.
• Releasing a hold clears status_reason to NULL, so any earlier reason on a running row (for example approval_granted) is not restored.
• The same stale-sweep gap exists on main.

Tests

The new tests are internal/handlers/offline_hold_test.go and internal/storage/offline_hold_test.go, run against real SQLite:

• An opted-in node offline past the stale timeout keeps its execution running on both tables through repeated sweeps. A non-opted node in the same sweep is timed out.
• When the node comes back with the execution listed as inflight, the hold is released, and its later succeeded status report lands normally.
• When it comes back without the execution, the execution is failed as node_lost_execution, with that message.
• When the node is removed, its held execution is failed as node_removed.
• When the ceiling passes, the held execution is failed as node_offline_too_long.
• An opted-in node that is online but stops listing an execution is still timed out.
• The storage tests show that all three sweeps skip held rows, and that a hold takes only the quiet rows of the named node.

Focused runs passed for ./internal/handlers (cleanup, lease, inflight, orphan and offline-hold tests), ./internal/storage (stale, orphan and hold tests), ./internal/config, ./internal/server (config) and ./internal/nodelink.

Engine side

The engine needs a one-line change in RegistrationFor (internal/factory/agentfield/registration.go). Add "custom": {"holds_executions_offline": true} under metadata. The existing codeaf key belongs in custom too, because the CP's AgentMetadata only keeps deployment, performance and custom, so a top-level codeaf key is dropped today.