[ycarogabriel-lgtm]

What and why

On an M1 (16 GB), Ocarina of Time (0400486024DEC000) hangs the GPU from time to time after an area change: kIOGPUCommandBufferCallbackErrorHang, fw_ta_state 1 (the tiler). After 2 to 5 such hangs macOS revokes the process's GPU access. The hung command buffers always hold draws of the Hi-Z task a5288d0a6c4971fd with its mesh 3439aaa071d868ae (cliffs and giant trunks); see #1326 and #1605.

With DM_NVN_ISOLATE_TASK=1, each draw of that task goes in a command buffer of its own. The render pass ends and the segment is committed right before and right after the draw, through the same sub_cut a buffer → texture copy forces. =<keys> names other task shaders. It is off by default.

This is a workaround, not a fix: it kept the hang from happening on our machine, but the cause inside the tiler is still unknown.

Evidence (M1, 16 GB, macOS 26.6.2)

Measured on our integration branch, which is older than this main and carries ports of #1707, #1708, #1720 and others. The change applies cleanly here, but these runs were not repeated on this base.

• Without isolation, scripted 300-420 s walks through an area change hung in most runs (our estimate, about 15 of 16; not a controlled count).
• With isolation, 4 of 4 walks of 420 s ran without a hang. Two later 300 s validation walks also had no hang. Our test log: EXP-meshM1-51..54, 57, 59.
• GPU cost: +0 to 2 ms a frame before the area change, nothing measurable after it.
• Bisection of the mesh, all with diagnostic switches on the same walk:
  • With the mesh body fully running but emitting no primitives, it did not hang (3/3).
  • Degenerating every primitive with a w ≤ 0 corner did not help.
  • Outputs, reads, compiler options, residency, barriers, pool table and pool chunking were each ruled out one at a time.
  • So the hang needs the mesh to emit triangles. That the task's draws sharing a command buffer with the rest of the pass matters is our inference from the cut working, not something we measured directly.
• A similar isolation result is reported on an M2 Pro in Raw global memory through a pointer read when the draw is bound: the host fills each fixed-offset pointer's pool range before the draw (chain 1ffcb6b77740) #1585; we have not run their code.
• The game runs smoothly for ~7-8 minutes in Kokiri Forest, #1828 (M2 Pro) reports that DM_NVN_CUT_BUSY=0 alone does not prevent it.

How to reproduce

1. Ocarina of Time v131072, a scripted walk of 300-420 s from a Kokiri Forest save that crosses an area change.
2. Run without the switch: we usually see an ErrorHang, and macOS revokes the GPU after a few of them.
3. Run again with DM_NVN_ISOLATE_TASK=1. The log prints task isolation: N draws isolated, M cuts taken before them every 20,000 draws.

Risks and caveats

• Off by default, so nothing changes unless the switch is set.
• Each cut ends the render pass and commits a segment. On a draw-heavy pass that costs load/store traffic and submissions; we measured only up to 2 ms a frame, on one machine.
• Not root-caused. Ocarina of Time (0400486024DEC000 v131072, Mac): GPU hang about 10 minutes into play takes WindowServer down #1326 reads a5288's only loop as a Hi-Z scan bounded by c3[193].xy. A separate diagnostics PR logs those values per draw, so a hung pass can be compared with the ones that pass.

Checked

• Note: the message of the first commit (9685e93) still mentions a local hang guard that does not exist in this repo; the next commit fixes the code comment. A squash on merge would clean that up.
• Host builds; no new compiler warnings.
• t_pso_archive with MTL_DEBUG_LAYER=1: 135 checks, 0 failed.
• check_t_geometry_mesh: all three variants pass; t_geometry_mesh also passes with DM_NVN_ISOLATE_TASK=1.
• t_mesh_first_use: 42 checks, 0 failed, both off and on.
• test_env_switches: the new row is accepted. The test still fails on DM_NVN_BC_ALIAS and DM_YIELD_CLOCK, which also fail on main 7f50222.

🤖 Generated with Claude Code

[Mkhurana3]

Tested this on a second chip: M2 MacBook Air, 16 GB. The diff applied cleanly on main 1794a35 (one hunk with an 8-line offset in nvn_geometry.inc), and DM_NVN_ISOLATE_TASK=1 does what the description says.

Trigger: the autosave just before Saria gives Link the ocarina, the repeatable hang from #1690 / #1326 (6 of 6 runs dead within 158-241 s on this build without it). With the switch on, nothing skipped, render scale 1.0, MetalFX present upscale: the Saria cutscene played through and I played Hyrule Field for about 8 minutes. The log has no hang, no ErrorHang and no GPU-access-revoked line, and the isolation counter reached 100,001 isolated draws (200,001 cuts).

One run only, so not conclusive. I have no frame-time numbers for the extra cuts yet (scale 1.0 was too heavy for an Air to play smoothly in Hyrule Field, so I will measure at a lower scale). Details and the full comparison are on #1326. I will post a longer run here.

[ycarogabriel-lgtm]

@Mkhurana3, thank you for trying it on a second chip and for the careful caveats.

What we have on the switch, from one MacBook Pro M1 (16 GB, macOS 26.6.2) on an older main: four scripted 420 s walks through an area change with DM_NVN_ISOLATE_TASK=1 ran without a hang (without it, about 15 of 16 hung by our estimate, not a controlled count); the cost measured before the area change was +0 to 2 ms of GPU a frame. A human session of about 55 minutes (about 100,000 frames) with the task isolated and the pipelines served from an archive had no ErrorHang and no revoke (details in my comment on #1326 at 01:50). An earlier human session of about 26 minutes on one of our builds with the isolation on ended in a page fault in an ordinary pass whose command buffer held no task/mesh draw, not in an ErrorHang. On the other side, lunks reports on #1326 that on an M5 Pro the hang still came with the task isolated, so a clean run is not the same on every chip.

If you have the time, what would help most is the same route with the switch off and on at one fixed render scale, with the frame time and GPU ms for each (and the hang count if one occurs), plus your chip and macOS version. That would show what the extra cuts cost on an M2, which we cannot measure. We make no claim about the cause or that this cures it.

[Mkhurana3]

A/B with the switch off and on, as asked. Chip: Apple M2 (MacBook Air), 16 GB, macOS 27.2. Build: main 1794a35 + this diff.

Setup: same route both times, render scale fixed at 0.75, MetalFX present upscale, 30 fps target. I loaded the autosave just before Saria and only watched the cutscene (no input), quitting at the same point both times (Link has the ocarina and the button prompt shows, before he leaves the forest). OFF ran about 3,600 frames, ON about 4,500. I compare the first 3,600 frames (12 windows of 300) in both.

Per-window p50 averaged over those 12 windows:

• GPU ms: OFF mean 47.7 (median 45.0), ON mean 52.2 (median 51.5), so about +4.5 ms a frame (+9%) with isolation on
• frame time ms: OFF mean 52.3, ON mean 56.6, about +4.3 ms

No ErrorHang and no revoke in either run, but both runs stopped before the area change, so this says nothing about hangs. The ON log counted 40,001 isolated draws.

Caveats: one pair of runs, played by hand, and the two are not frame-identical (the window-to-window spread is +-15 ms in both). The ON run had two windows around frames 1500-1800 with p95 GPU over 130 ms and one frame over 1 s (I have not looked into its cause); the OFF run had none. I would not read +9% as precise. At 0.75 neither mode holds 30 fps on this chip in that scene (GPU p50 about 35-70 ms in both), so the extra cuts land on a GPU-bound machine.

I can post the raw per-window tables (frame time and GPU p50/p95 per 300 frames) if they would help.

[ycarogabriel-lgtm]

@Mkhurana3, thank you for the A/B. Yes, the raw per-window tables (frame time and GPU p50/p95 per 300 frames, for both runs) would help, especially around frames 1500-1800 of the ON run where p95 GPU went over 130 ms and one frame took over 1 s: it would be useful to know whether that window also shows new pipelines being compiled in the log, since a one-off compile and an effect of the extra cuts look the same in a summary but would have different causes.

For comparison, on our M1 the cost we measured was +0 to 2 ms of GPU a frame before the area change, but that was on a different route and scale, so the two numbers are not directly comparable; yours at scale 0.75 on a GPU-bound Air is the more demanding case. We have no claim about why the M2 number is larger.

[Mkhurana3]

Thanks. Raw tables for both runs (Apple M2 Air, scale 0.75, per 300 frames; first column is the frame the window ends at; frame time p50/p95 ms | GPU p50/p95 ms):

OFF

 300  67.14 / 85.00 | 64.60 / 82.26
 600  54.94 / 73.48 | 47.72 / 70.85
 900  74.37 / 109.37 | 68.73 / 76.63
1200  57.09 / 76.25 | 54.66 / 72.10
1500  46.77 / 53.84 | 41.90 / 48.17
1800  53.39 / 65.38 | 48.70 / 60.73
2100  45.10 / 48.55 | 40.64 / 43.85
2400  46.34 / 49.07 | 41.62 / 44.26
2700  43.47 / 46.54 | 39.11 / 41.55
3000  39.47 / 49.89 | 34.25 / 42.32
3300  48.57 / 57.92 | 43.70 / 52.89
3600  50.65 / 57.38 | 46.22 / 52.31

ON

 300  68.50 / 82.61 | 66.68 / 79.79
 600  57.81 / 68.62 | 55.99 / 66.44
 900  57.22 / 58.58 | 55.26 / 56.46
1200  57.48 / 60.32 | 55.67 / 58.36
1500  66.17 / 149.95 | 57.66 / 135.31
1800  92.71 / 144.31 | 87.92 / 137.82
2100  53.51 / 65.99 | 47.66 / 58.92
2400  54.57 / 66.27 | 47.46 / 61.30
2700  46.09 / 50.48 | 41.93 / 46.87
3000  44.95 / 49.10 | 40.17 / 44.16
3300  42.66 / 47.41 | 38.19 / 42.94
3600  37.04 / 41.49 | 32.35 / 36.46
3900  44.84 / 54.25 | 40.11 / 49.42
4200  50.33 / 54.43 | 46.10 / 50.09
4500  48.84 / 53.95 | 44.46 / 49.29

On whether the two high-p95 ON windows show new pipelines, from the log's per-300-frame pipeline counters and its [longframe] lines:

• Window ending at frame 1500 (frames 1200-1500, p95 GPU 135 ms): yes. 9 pipeline bases were compiled (3,244.8 ms summed over all threads, 49.7 ms of it on the submit thread) and 18 archive lookups found nothing. Frames 1311 and 1322 took 402 and 453 ms, with submit+present at 395 and 403 ms and GPU at 20 and 94 ms.
• Window ending at frame 1800 (frames 1500-1800, p95 GPU 138 ms): no. 0 pipelines compiled and 0 ms of pipeline work on the submit thread. Its slow frames are GPU-heavy: frame 1501 took 221 ms (GPU 139 ms) and frame 1535 took 208 ms (GPU 116 ms), both with 0 new pipelines. So I can say the first window has compiles and the second has none, but not what makes the second one slow.
• The OFF run also compiled in its window ending at frame 1500 (2 bases, 471.9 ms, 52.5 ms on the submit thread), and had two frames of about 430-450 ms with submit+present near 406 ms (frames 461 and 472).

The one frame over 1 s in the ON run is frame 390, 56 s in, not in these windows: whole frame 20,498 ms with submit+present 0.2 ms, GPU 76 ms and no new pipelines, so the time was not spent in submit, present or the GPU. I have not found what it was.

The two runs were not frame-identical (the ON run is longer, and ON compiled 9 bases to OFF's 2), so the +4.5 ms is not a clean measure of the extra cuts. Window by window (ON minus OFF, GPU p50), the other ten windows sum to about +0.1 ms (about 0.01 ms each); the whole difference comes from the two slow ON windows, ending at 1500 (+15.8 ms) and 1800 (+39.2 ms), and only the first of them compiled. Without them I see no cost from the extra cuts in this pair, but this is still one pair of hand-played runs, and a run with matching compile activity would settle it.

[ycarogabriel-lgtm]

@Mkhurana3, thank you for the raw tables and for going through the log window by window.

Reading your numbers: the other ten windows differ by about +0.1 ms of GPU p50 in total, and the whole +4.5 ms comes from the two slow ON windows, of which only the one ending at frame 1500 shows pipeline compiles (9 against 2 in OFF). So in this pair the cost of the extra cuts cannot be separated from compile activity, as you say.

What we can add is little: we have no M2 data, and nothing measured that explains the window ending at frame 1800 (slow, 0 compiles) or the 20.5 s frame 390 with 0.2 ms in submit+present. Our own M1 figure (+0 to 2 ms a frame before the area change) comes from a different route and scale and does not bear on those. A matched pair with the same compile activity, as you suggest, is what would settle it; we make no claim from this one.