Add support for replay tests#324849
Draft
pwang347 wants to merge 3 commits into
Draft
Conversation
A wire-agnostic CAPI record/replay proxy (capiReplayProxy.ts) sits in front of the mock LLM server behind the agent host's existing env seam. It captures raw model responses to committed JSON fixtures and replays them deterministically with zero upstream contact - exercising the entire agent host e2e (real server, real bundled Copilot SDK/CLI subprocess, real JSON-RPC/AHP protocol) without a token. Tests (copilotReplay.integrationTest.ts): - single-turn, multi-turn, and no-upstream round-trip tests - a scripted shell tool call: a mock scenario drives the SDK to emit a bash tool_use over the wire, which the agent host executes end-to-end (toolCallStart) and the two-call sequence replays deterministically. Exercises the tool-call parsing/execution path a CLI/SDK bump is most likely to regress. - startRealServer gains 'capiReplay' + 'mockScenarios' options; IServerHandle exposes the proxy. - Fixtures under captures/copilotReplay/ (paths normalized, no secrets). - Record with AGENT_HOST_REPLAY_RECORD=1; replay (default) is strict so an unexpected SDK call after a bump fails the run.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a record/replay testing harness for the agent host's real-SDK integration tests. Previously these provider suites (Copilot, Claude, Codex) were gated behind AGENT_HOST_REAL_SDK=1 and required a live GitHub token plus network access, so they were effectively disabled in CI. The PR introduces a wire-level CAPI record/replay proxy that captures model traffic into committed YAML fixtures and replays them deterministically (tokenless, no network), letting the shared suite run by default. New fixtures are re-recorded against real CAPI via AGENT_HOST_REPLAY_RECORD=1.
Changes:
- New
CapiReplayProxythat sits in front of CAPI/GitHub upstreams, records raw responses, and replays them by(method, path)sequence; pluscapiWireCodec(Anthropic Messages + OpenAI Responses SSE parse/regenerate) andcapiStubs(hardcoded ancillary bootstrap endpoints kept out of fixtures). - Wires
capiReplayintostartRealServer/shared test helpers, enables the provider suites by default in replay mode, and commits per-test YAML capture fixtures. - Adds
js-yaml/@types/js-yamlas direct devDependencies (lockfile pinsjs-yaml@4.1.1).
Show a summary per file
| File | Description |
|---|---|
capiReplayProxy.ts |
New record/replay HTTP proxy: fixture I/O, normalization, secret/model redaction, strict cache-miss check. |
capiWireCodec.ts |
SSE aggregation/regeneration and readable request summaries for Anthropic and Responses dialects. |
capiStubs.ts |
Hardcoded stubs for ancillary CAPI/GitHub bootstrap endpoints served on replay. |
testHelpers.ts |
Threads capiReplay/mockScenarios options through startRealServer; routes agent host to the proxy. |
realSdkTestHelpers.ts |
Record/replay helpers, tokenless replay path, teardown flushing, deterministic sentinel. |
copilotRealSdk/codexRealSdk/claudeRealSdk.integrationTest.ts |
Enable suites by default in replay; gate non-deterministic Codex steering behind a real-run flag. |
captures/realSdk/*.yaml |
Committed per-test capture fixtures. |
package.json / package-lock.json |
Add js-yaml/@types/js-yaml devDependencies; lock downgrades js-yaml to 4.1.1. |
Review details
- Files reviewed: 31/32 changed files
- Comments generated: 5
- Review effort level: Medium
| "istanbul-lib-report": "^3.0.0", | ||
| "istanbul-lib-source-maps": "^4.0.1", | ||
| "istanbul-reports": "^3.1.5", | ||
| "js-yaml": "^4.1.1", |
| ${workdir}/ahp-subagent-replay-D1IH5t/file-b.txt | ||
| - type: tool_result | ||
| tool_use_id: toolcall_3 | ||
| content: "Shell ID: 013f779e-e215-4b2d-b431-f662bacb1be2\nExit code: 0\nls -la /var/folders/yj \r/46gvyjxx5479vlbnxz_914xw0000gn/T/ahp-subagent-replay-D1IH5t\r\ntotal 16\r\ndrwx------ 4 paulwang staff 128 Jul 7 16:05 .\r\ndrwx------@ 975 paulwang staff 31200 Jul 7 16:06 ..\r\n-rw-r--r-- 1 paulwang staff 5 Jul 7 16:05 file-a.txt\r\n-rw-r--r-- 1 paulwang staff 4 Jul 7 16:05 file-b.txt" |
Comment on lines
+68
to
+71
| drwx------ 4 paulwang staff 128 Jul 7 16:13 . | ||
| drwx------@ 975 paulwang staff 31200 Jul 7 16:13 .. | ||
| -rw-r--r-- 1 paulwang staff 5 Jul 7 16:13 file-a.txt | ||
| -rw-r--r-- 1 paulwang staff 4 Jul 7 16:13 file-b.txt |
Comment on lines
+87
to
+90
| drwx------ 4 paulwang staff 128 Jul 7 16:13 . | ||
| drwx------@ 975 paulwang staff 31200 Jul 7 16:13 .. | ||
| -rw-r--r-- 1 paulwang staff 5 Jul 7 16:13 file-a.txt | ||
| -rw-r--r-- 1 paulwang staff 4 Jul 7 16:13 file-b.txt |
Comment on lines
+359
to
+361
| const body = item.dialect === 'responses' ? responsesMessageToSse(item.message) : anthropicMessageToSse(item.message); | ||
| res.writeHead(200, { 'content-type': 'text/event-stream', 'cache-control': 'no-cache' }); | ||
| res.end(body); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.