[JioChoi]

What and why

On an 8 GB M1 MacBook Air, PRAGMATA (0400C76024710000) runs out of GPU memory in its opening load, after New Game. The residency set reaches 5.2 GB of Metal's 5,461 MB working set. Command buffers fail with kIOGPUCommandBufferCallbackErrorOutOfMemory and the submit thread waits in IOGPUCommandQueueSubmitCommandBuffers. Twice, WindowServer hit its watchdog and the desktop session restarted.

What filled the set at that point: 2.45 GB of textures and 2.37 GB of shared buffers.

• All of PRAGMATA's ~15,500 registered textures are needed, because its shaders read 2D textures through the whole texture pool.
• The shared buffers covered only 1.9 GB of distinct pages.

The goal is the iPhone 15 Pro, which also has 8 GB. With #1315's guest file the footprint stays at 3.2–3.6 GB, so GPU memory is what is left to fit.

Changes, all in nvn_mtl.m:

• Replaced chunks retire (big_lookup). A large-pool chunk that a longer one replaces is retired once the submissions that may use it complete, as a freed pool's buffer is. Before, it stayed resident until the pool went. Every cache of pool addresses checks g_pool_gen, which big_new bumps.
• Pools from 128 MB are chunked (was 512 MB; BIG_MAX 32 → 64). PRAGMATA's 256–300 MB pools were wired whole for any binding.
• The pool walk checks the type first. A walk for handles traced to no slot skips IDs whose descriptor cannot be of the walked type (ttype_apart). It used to make a Metal texture for every registered ID before it compared the type.
• DM_NVN_TEX_MIP_SKIP=<px> (1 = 512; off by default). CPU-written block-compressed 2D, 2D array and cube textures with a mip chain, both sides at least that size, are made without their finest level: a quarter of the memory. Views, copies, CPU readbacks and a sparse texture's level clamp take the guest's level less the skip (tex_mip_skipped, lv_host). Render targets and shader-written textures are never in these formats. A shader that asks a texture's size gets the halved one.
• DM_NVN_GMEM_RES=0. Pools reached only through the raw global memory table stay out of the residency set until a binding addresses them, and large pools stay in chunks instead of one whole-pool entry.
• DM_NVN_DESWZ_GPU=0. Swizzled textures are deswizzled on the CPU.
• Diagnostics. The out-of-memory report also gives the distinct pages under the shared buffers and texture memory by pixel format. DM_NVN_RES_LIMIT_MB=<MB> stops the game gracefully, with that report, once the set passes that size. I used it to keep testing without taking WindowServer down.

Checked

• Game run: PRAGMATA v196608 on the MacBook Air M1 (8 GB), with the PRAGMATA bring-up changes from my working tree (NVN2 push blocks and shader work, not in this PR) and host: the guest's memory in files (DM_GUEST_FILE, DM_GUEST_FILE_CHUNK) #1315 applied. Settings: DM_GUEST_FILE=1 DM_GUEST_FILE_CHUNK=512M DM_NVN_GMEM_RES=0 DM_NVN_DESWZ_GPU=0 DM_NVN_TEX_MIP_SKIP=1.
  • At the title, the set fell from 3,985 MB to 3,241 MB (textures 2,308 → 1,812 MB). The title screen looks as before.
  • Through the opening load, the set peaked at 4,284 MB with no GPU errors, and the game reached its opening scene. Before, it hit Insufficient Memory at about 5.2 GB.
  • Still open: that scene takes 250–290 ms of GPU time a frame on the M1.
• test_docs.py, test_env_switches.py: pass.
• Not run: regress.py check --games (no regression harness or other game builds on this machine), and the host and gfx unit tests. Everything new except the pool changes is behind switches that are off by default. The pool changes (retiring a replaced chunk; chunks from 128 MB) apply to every game.

Docs

• docs/reference/memory.md: "GPU out of memory on an 8 GB Mac: the residency set passes Metal's working set". Symptom index regenerated (docs_symptoms.py).
• docs/env_switches.tsv: DM_NVN_TEX_MIP_SKIP, DM_NVN_RES_LIMIT_MB, DM_NVN_GMEM_RES, DM_NVN_DESWZ_GPU.
• No game files, keys or data from a game in the change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DeVTtdCXD7t64Ek6e1py6A

[Th0m45k]

A data point from an 8 GiB iPhone, in case it helps: this PR on an iPhone Duo (iPhone19,4, A20 Pro, iOS 27.2, per-process limit 8,191 MB), with Mario Kart World 1.0.0 (base v0, handheld).

Build: this PR's head (fc815c3), with #1315 (guest file) and #1306 (env.txt samplers) cherry-picked on top. Signed with Increased Memory Limit + Extended Virtual Addressing. Documents/env.txt: DM_GUEST_FILE=1, DM_GUEST_FILE_CHUNK=1G, DM_NVN_GMEM_RES=0. I left DM_NVN_TEX_MIP_SKIP and DM_NVN_DESWZ_GPU unset.

Route: the same Grand Prix (4 races, ~19 min) as in the earlier runs, after a full app restart each time. Numbers come from the log's memory line, [longframe] lines, and DM_MEMSTAT (once a second).

	before (main + hand-merged guest file, view cache 64)	this PR + #1315, GMEM_RES=0
frames > 50 ms with no new pipeline, first 45k frames	70 / 74 (two runs)	14
frames > 50 ms with a new pipeline, same window	18 / 24	14
GPU allocated at the end	10.1 / 10.0 GB	8.3 GB
graphics ledger in the footprint, max	4.04 / 3.93 GB	3.92 GB
memory pools, resident at the end	3.69 / 3.80 GB	3.87 GB
least headroom to the limit	3.42 / 3.55 GB	3.48 GB

The player reported races 1–3 at a steady 60 fps and only occasional spikes in race 4, the smoothest run so far. Before, the spikes started in race 2. I saw no visual problems, and there were no new [RED] lines; the two present (a VP9 packet refusal and one swizzled texture outside a pool) show up the same way in every earlier run.

My reading, not measured directly: on this phone the change mostly removes stalls, not footprint. The graphics ledger peaks where it did before, while GPU allocated and the stalls both drop, which fits pools being wired in chunks instead of whole.

Caveats: this is one run per configuration. I did not separate the pool changes from DM_NVN_GMEM_RES=0. The baseline also differs in its guest-file implementation (a hand merge of the switch2ios commits instead of #1315) and in having a larger view cache (DM_NVN_VC_WAYS=64). Happy to run a split (with and without GMEM_RES=0, or TEX_MIP_SKIP=1) if that is useful.

[bettterlabs]

Tried this on Ocarina of Time (0400486024DEC000 v131072) on a Mac mini M4, 16 GB, macOS 27.2. The base is older: main 7550be1 + #1299 at 6516feb + #1312, with this PR cherry-picked. It conflicts with #1299 in pool_gpu (#1299 moves the loop body into pool_entry) and in gmem_map (#1299 returns the buffer). Handheld, DM_PSO_MTL4=0 (the pipeline archive sits on an external drive here, see #1265).

Four runs, one per setting, all from the same Kokiri Forest save. After the load I stayed in Link's bedroom in every run: standing still, then turning the camera, then jumping around, and near the end looking out of the front door or the back window. No area change. A guard stopped the runs with this PR on Mac free memory. The run from before this PR went until it froze.

After the save loads	before this PR	both switches	DM_NVN_TEX_MIP_SKIP=1	DM_NVN_GMEM_RES=0
fps per 300 frames	35–46	30–41	34–47, then 25 → 16 ¹	17–27
frame time p95	29–38 ms	30–43 ms	26–32 ms, then up to 200 ms ¹	63–215 ms
GPU time p50	28–36 ms	27–50 ms	28–41 ms	40–61 ms
GPU waits so far	72 (144 after the out of memory)	53	12	1,937
pools resident	all 4,481	~1,190 of 4,524	all	~1,190 of 4,523
GPU allocated, title screen	5.1 GB	1.8 GB	5.4 GB	1.8 GB
residency set, after the load → end	9.1 → 11.9 GB	7.7 → 10.8 GB	9.5 → 11.3 GB	7.7 → 10.7 GB
footprint, after the load → end	10.0 → 12.7 GB	9.9 → 13.0 GB	10.2 → 12.0 GB	10.0 → 13.1 GB
how it ended (time after the load)	GPU out of memory (113 s)	guard ² (100 s)	guard (118 s)	guard (155 s)

¹ From about 45 s after the load, with the Mac mostly at 6–9% free and GPU waits still at 12: system memory, not the GPU.
² At 8% free; the other two at 5%.

Before this PR, the bedroom alone fills Metal's working set. 113 s after the load, with no area change, the residency set reached Metal's limit, and frames took 0.4–1.2 s until I closed it:

[RED] GPU out of memory: the residency set holds 11569 objects, 11897 MB (Metal's recommended working set 12124 MB, GPU allocated 11731 MB): textures 7021, 6242 MB; heaps 0, 0 MB; shared buffers 4541, 5463 MB; private buffers 7, 192 MB; the largest:
  buffer 0x7147700000+0x98000000 (shared): 2432 MB
  buffer 0x70c5c14000+0x13104000 (shared): 305 MB
  texture 12288x12288x1 format 250 levels 1: 296 MB

That's the same 2,432 MB graphics pool, resident whole, as in nafields's report on #1265. This run also logged a kIOGPUCommandBufferCallbackErrorHang a second after the pool table overflowed, as david-on-spring describes on #1347. None of the runs with this PR logged a hang or an out-of-memory error, though their pool tables overflowed too.

DM_NVN_GMEM_RES=0 buys headroom. Alone, it cost GPU time here.

• About 1,190 pools resident instead of all of them. GPU allocated on the title screen is 1.8 GB instead of 5.1–5.4 GB, and the residency set is 1.4–1.8 GB smaller right after the load. Neither run with it reached Metal's limit before the guard, at 10.7–10.8 GB.
• Alone, GPU time rose by about 10–25 ms a frame, and the submit thread waited for a free command allocator 1,937 times, against 12–72 in the other runs. Those waits are the 150–300 ms frames. With DM_NVN_TEX_MIP_SKIP=1 added, waits fell back to 53 and frame times came close to the baseline.
• I can't tell from the log why. Residency commits per 300 frames were 1,400–1,560 with it alone, 1,200–1,270 with both, 1,130–1,340 with DM_NVN_TEX_MIP_SKIP=1 alone, and 610–660 before this PR.
• If it does the same in PRAGMATA, it may be part of the 250–290 ms of GPU time you noted for its opening scene; an A/B with and without it there would tell.

DM_NVN_TEX_MIP_SKIP=1 alone looked like the baseline in frame time and memory. The log doesn't say how many textures lost their finest level, so I can't tell how much of this game it reaches; a count and MB in the log would show it. The DM_NVN_GMEM_RES=0 run, which didn't have it, may have looked a little sharper to me than the run with both.

The footprint doesn't change with any of it. In every run it was about 3.8 GB on the title screen, about 10 GB right after the load, and 12–13 GB 1.5–2.5 minutes later. Footprint and GPU allocated rose by about the same 1.8–3.1 GB in every run while the pool count stayed flat, so the growth is host GPU memory. The log's shared-texture count also doubled (583 → 1,154 in the baseline). It rose through the whole run, not only once I looked outside. So on a 16 GB Mac, once the working set has headroom, system memory runs out at about the same point. This log doesn't say whether the new textures are host copies of the game's textures (#1281) or render targets, which @david-on-spring found most of the GPU memory to be at a scene change. Their 15 minutes at 10.4 GB on #1347 included their pool-table changes, which this build doesn't have; their ~40 ms a frame is about what both switches gave here.

The pool table overflowed at the load in every run. RED_ONCE logs only the first count, 1–7 here.

Single runs with my own hands on the controller, so treat small differences as noise. The large ones, the allocator waits and the out-of-memory, are well outside it. If a by-format breakdown would help, I can rerun with DM_NVN_RES_LIMIT_MB to get this PR's report at a set size.

[JioChoi]

Closing in favour of the draft PRs #1614 (large pools held once in the residency set, stacked on #1407) and #1615 (GPU memory guard measure, stacked on #1392). pool_reuse_dead was wrong, as #1407's review found: a no-copy buffer keeps the pages it was made over. DM_NVN_RES_LIMIT_MB moved into #1391's guard, and the mip skip continues in #1418.