Skip to content

bench.py --verdict: a long-play check per run (frames of 1 s or more, event-wait holds, slow 1000-frame stretches) - #2524

Open
dpadr wants to merge 1 commit into
seedds:mainfrom
dpadr:bench-verdict
Open

dpadr wants to merge 1 commit into
seedds:mainfrom
dpadr:bench-verdict

Conversation

@dpadr

@dpadr dpadr commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Frame-time percentiles over a long window hide a slow stretch. In a 6-minute run, two 1000-frame stretches at 70–90 ms move p50 and p95 only a little, while the player sees about a minute of stutter. This adds what a long run needs, for any game:

  • Per-run counts: every run's summary now has long_frames (frames of 1 s or more), holds (a GPU queue held by an event wait, the host's an event wait has held it line), and the median of each stretch of --stretch frames (default 1000), with worst_stretch. results.md shows long_frames, worst_stretch and holds.
  • --verdict: judges each run CLEAN or BAD on those counts. A stretch is slow when its median is over --slow-factor (1.5) times the run's own frame p50, or over --slow-ms when given. The relative default works across games and frame targets.
  • Report and exit code: results.md gets a verdict line, and the exit code is 1 when any run is BAD. Without --verdict, the exit code and the existing columns are unchanged.

bench.summary() keeps its signature (a new optional stretch argument), so scripts/regress.py's call is unchanged.

What it catches. I applied the branch's summary and judge to the frames.tsv and log of long runs of Ocarina of Time on an M2 Air (a 12,000-frame Kokiri walk, judged from frame 1000):

run frame p50 verdict why
plain main ccd834e + haze fix, run 1 48.8 BAD 2 of 11 stretches over 73 ms (worst 91 ms)
plain main ccd834e + haze fix, run 2 45.4 BAD 1 of 11 stretches over 68 ms (worst 77 ms)
a build with a signal-cut revert 49.5 BAD 97 frames of 1 s or more (worst 23.5 s), 8 event-wait holds
a build with fences stripped 33.2 BAD 2 of 11 stretches over 50 ms
the reference build, 4 runs 27.6–30.3 CLEAN

These agree with the absolute rule we used for those runs (no 1000-frame median over 40 ms); --slow-ms 40 gives that rule.

Tests (dm/tools/tests/test_bench.py):

  • test_stretches_and_judge: stretches, the half-stretch tail, the factor and absolute limits, holds, a 1.2 s frame, no frames.
  • test_verdict_cli: a fake host writes a slow stretch and a hold line → BAD in results.md and exit 1; a clean run → exit 0.
  • test_bench.py: 9 tests OK. test_regress.py: 65 OK (Python 3.14; on 3.9 one test fails on main too, shutil.rmtree(onexc=)). test_docs.py OK.

Not included: the GPU watchdog line (#1550, not on main). When it lands, add its line to the holds count.

Regress: bench.py is a measuring tool. It changes no chain, host or rewriter, and regress.py's call into bench.summary is unchanged and tested, so no game run is needed.

🤖 Generated with Claude Code

… event-wait holds, slow 1000-frame stretches)

Each run's summary now counts frames of 1 s or more (long_frames), the event-wait holds in its log (holds) and the median of each stretch of --stretch frames
(stretches, worst_stretch); results.md shows long_frames, worst_stretch and holds. --verdict judges every run CLEAN or BAD on them: a stretch is slow when its
median is over --slow-factor (1.5) times the run's frame p50, or over --slow-ms. results.md gets a verdict line; exit 1 when any run is BAD.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added tools Tools and scripts (dm/tools, scripts) docs Documentation labels Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants