[PromethOps]

What and why

A game may leave the descriptors of the textures it finalized registered in its texture pool. Every pool walk resolves them, which kept their shared textures (g_alias) and refreshed their last-use stamp. A shared texture left the table only when its pool was finalized or another texture took over its storage. A game that keeps its pools and loads the next scene elsewhere in them does neither.

The Legend of Zelda: Ocarina of Time (Switch 2, 0400486024DEC000 v131072, profile in #1299) keeps about 12,000 such descriptors. After the talk with Saria in Kokiri Forest, 2,000 to 2,500 shared textures lost their live object at once. Textures went from 5 GB to 10–16 GB, and the residency set passed the 18.2 GB working set of a 24 GB M5 Pro. The GPU then faulted (PageFault) or hung, macOS ignored every later submission, or the submit thread blocked in the kernel with the Mac frozen.

alias_retire_idle (nvn_mtl.m, at the start of each submission) runs when the residency set holds more than 85 % of the working set. It retires shared textures that no live texture object (or a view one holds) resolves to and that no lookup for a live object has used for 64 submissions, the longest idle first, until the set is down to 75 %.

• Only lookups for live objects count as use (g_alias_lseen). Reclaiming by any lookup keeps every texture a stale descriptor names. DM_CTL poolstat and texstat, and what fills Ocarina of Time's GPU working set (iPad out of memory at 8,259 of 8,192 MB) #1436 notes that reclaiming by lookup idleness alone retired targets in use.
• A finalized descriptor over a retired key resolves to nothing, the pool table's dummy. Making them again brought the old scenes back by the thousand (19,000 made in 300 frames), too new to retire.
• A live object over a retired key gets a new texture filled from its storage (tex_init_fill).
• DM_NVN_ALIAS_RETIRE=0 turns it off.
• DM_NVN_WS_MB sets another working set for it and for restop, to try a smaller Mac's budget on a larger one: a 16 GB Mac recommends about 10.7 GB.

A game within its budget is unchanged: nothing is retired below 85 %.

This is complementary to #1393 (pool_skip_retired, a profile key that gives finalized IDs the dummy) and #1404. Here the rule is general, and it acts only under memory pressure, on textures no live object holds.

Measured

Ocarina of Time, 24 GB M5 Pro, repeated replays of the scene after Saria:

	before	after
textures after the scene change	10–16 GB	about 6 GB
result	GPU fault, hang or frozen Mac	no error over the replays

• DM_NVN_ALIAS_RETIRE=0 on the same replay: 16.6 GB of textures and a freeze.
• With DM_NVN_WS_MB=10700 (a 16 GB Mac's budget) the game stayed within it.

Checked

• Games run, with title ID and version: The Legend of Zelda: Ocarina of Time (Switch 2, 0400486024DEC000, update v131072, SDK 22.2.8), with the profile from [WIP] The Legend of Zelda: Ocarina of Time (Switch 2, 0400486024DEC000, v0, SDK 22.2.8) runs on the Mac through the title movie into Kokiri Forest #1299 — Kokiri Forest, the talk with Saria, the Great Deku Tree in and out, several sessions of 5 to 15 minutes; compared with restop / the memory report lines, frame times and screenshots.
• Devices and OS: MacBook Pro M5 Pro, 24 GB, macOS 27.2.
• make -C dm/gfx/tests GAME=<id> check_t_alias_retire (new): 6 checks with direct and deferred encoding, and 3 with DM_NVN_ALIAS_RETIRE=0. It covers:
  • a held texture stays;
  • an idle unreferenced one is retired;
  • its finalized descriptor resolves to nothing;
  • a live object over it gets a new texture.
• t_image_alias (318), t_cpu_storage (32), t_cpu_tex (20), t_linear_readback (24), test_docs.py.
• Not checked: scripts/regress.py check --games on other games; only Ocarina of Time was available here. The change only acts above 85 % of the working set, so games within their budget should see none. A game that keeps drawing with finalized descriptors for long under memory pressure would read the dummy for them; the reference entry lists this limit.

Docs

• The finding is in its docs/reference/rendering.md entry, and python3 dm/tools/docs_symptoms.py was run (the symptom index)
• A chain tool changed: not the case (no chain tool is changed)
• No game files, keys or data extracted from a game in the change or the attachments

[PucaVaz]

Tested on main 75c1a8c, with #1453, #1332 and #1446 also applied, on an M4 MacBook with 24 GB (Metal's working set 18,186 MB), from the Kokiri Forest autosave at Link's house, in two scripted runs: 10 minutes in the house; and from a fresh load, out of the door, through Saria's greeting and 10 minutes of walking around the house.

• No freeze, hang or GPU error after the talk with Saria.
• alias_retire_idle retired nothing: the residency set stayed at 4.9–5.4 GB, below the 85 % trigger (15.5 GB here). The end of the run had 1,502 shared textures, 639 of them unreferenced.
• So on today's main this Mac didn't reach the state the PR describes: the runs show neither the fix working nor a cost from it. t_alias_retire: 6/6 in both encoding modes, 3/3 off.

[PromethOps]

Thanks @PucaVaz, that is a useful result.

Your runs agree with the PR's design: on today's main the residency set stays at 4.9–5.4 GB, far below the 85 % trigger, so nothing is retired and there is no cost. The freeze after Saria was measured before main's Ocarina changes (raw pool table, textures made only when read, and the rest), which keep that scene well within the working set now. On main this PR no longer fixes a freeze you can reproduce. It is a safety net for a game, or a smaller Mac, that does reach the working set with textures nothing live holds.

The build check failure was mine. Two other gfx tests (t_direct_upload, t_view_swizzle) called tex_init_fill with its old two arguments. Fixed in 74c5abf; every gfx test builds, and the two pass (26,384 and 554 checks). The other failing checks fail the same way on main 75c1a8c.

I will update the description to say this plainly. If you would rather not carry a safety net that main does not currently need, closing it is fine with me.

[PromethOps]

Closing this: on today's main, Ocarina of Time no longer comes near its working set after the talk with Saria (PucaVaz's runs and mine: residency about 5 GB, nothing retired), so there is nothing for this to catch. Main's own memory changes fixed the freeze at its source. Thanks for testing it.