Skip to content

perf(shaders): store stage memory bytes with atomic AND and OR instead of a compare-and-swap loop - #1503

Closed
Romen-Adama-Dev wants to merge 1 commit into
seedds:mainfrom
Romen-Adama-Dev:perf/shared-store-and-or
Closed

Romen-Adama-Dev wants to merge 1 commit into
seedds:mainfrom
Romen-Adama-Dev:perf/shared-store-and-or

Conversation

@Romen-Adama-Dev

@Romen-Adama-Dev Romen-Adama-Dev commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

What

The translator's shared memory byte store (SharedStore8, also the 16-bit SharedStore16) was a compare-and-swap loop on the word, from upstream Ryujinx. GenerateSharedStore now clears the field with an atomic AND and sets it with an atomic OR. Lanes writing other bytes of the same word commute with both, so the word ends as the loop left it, without retries. A chain change: translator rebuilt, dmsh_chain.lock and dmsh_chain.h updated (chain 26a894c0f40a), the StageMemory test checks the new form. DM_NVN_SKIP_MESH=1 drops every task / mesh draw (as DM_NVN_SKIP_FS drops draws by fragment shader): how the cost was isolated.

Why

A task or mesh shader writes its primitive indices and count as bytes (0x927 modes 0x80 and 0x88). The 32 lanes of a workgroup write neighbouring bytes, so four lanes contend for each word and the loop ran about four times a store, with a divergent retry each time. Ocarina of Time's foliage mesh shaders have ten such stores a lane, over tens of thousands of workgroups a frame.

Semantics: two lanes writing the same byte at once now get a mix of their values where the loop gave one of the two; the hardware's order between them is undefined as well. No shader seen writes the same byte from two lanes.

Measured

The Legend of Zelda: Ocarina of Time, Kokiri Forest, M3 Pro (18 GB), DM_NVN_GPUPROF=every:600. The pass after the opaque pass holds the foliage mesh draws; a render pass's start stamp is written after its vertex stage, where Metal runs the object and mesh stages, so the mesh work shows as the gap before that pass or as its duration.

Run Foliage pass Frame Internal resolution (the game's controller)
main (671c68e), as shipped 8.3-8.6 ms 25-28 ms, 36-40 fps 854x480
mesh draws dropped (DM_NVN_SKIP_MESH=1) - 18.4 ms 1440x810
mesh bodies emptied (groups still launched) ~0 16.3 ms 1440x810
this change, the forest's 39 mesh shaders rebuilt by hand 3.5-4.0 ms 20.7 ms, 48 fps 1440x810
this change, full chain rebuild on top of 0f510fd (chain 26a894c0f40a on 0f510fd; rebased on e0dc79c as chain 3938bdc7c7c0) 4.9-5.4 ms (a denser view: 815-824 draws in the pass before, against 409-425) 20.8 ms p50, 22.0 ms p95, 48 fps 1120x630

Every run follows the same scripted route from the same save; the last row comes from the packed app, with 10.0 GB resident and no new [RED] line. The empty-body run shows the cost was in the translated body, not in launching the groups; the remaining 3.5-4.0 ms is the body itself (zeroing of the stage memory, loads, the output copy).

Checks

  • DMSH_TEST_XLATE=dm/gfx/tools/dmsh_xlate_bin/dmsh_xlate python3 dm/gfx/tests/test_dmsh_ampere_mem.py: 64 ok (the byte store has the two atomics and no compare-and-swap).
  • python3 dm/gfx/tests/test_dmsh_meshstage.py: 26 ok. python3 dm/tools/tests/test_nsp_chain.py: 17 ok. python3 dm/tools/tests/test_docs.py: ok.
  • scripts/regress.py check --games: not run, this Mac has only Ocarina of Time. The change reaches every game whose compute, task or mesh shaders store bytes or half-words in shared memory; a run on the usual five would be welcome.

Docs

  • docs/reference/shaders.md: "Stage memory byte stores through a compare-and-swap loop" (Symptom / Cause / Fix / Check / Found on), symptom index regenerated.
  • dm/games/0400486024DEC000/README.md: a line under "Found on this game".

@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders game: 0400486024DEC000 A game labels Nov 6, 2026
javierdv7 pushed a commit to javierdv7/nxnative that referenced this pull request Nov 6, 2026
…#1462's CAS, seedds#1487, seedds#1329, seedds#1410, seedds#1411, seedds#1412, seedds#1357): lock and chain header; conflict leftovers removed from shaders.md and test_dmsh_ampere_alu.py
gnosticgeek pushed a commit to gnosticgeek/nxforge that referenced this pull request Nov 6, 2026
# Conflicts:
#	dm/gfx/nvn/dmsh_chain.h
#	dm/gfx/tools/dmsh_chain.lock
#	dm/gfx/tools/dmsh_xlate_bin/Ryujinx.Common.dll
#	dm/gfx/tools/dmsh_xlate_bin/Ryujinx.Graphics.Shader.dll
#	dm/gfx/tools/dmsh_xlate_bin/Spv.Generator.dll
#	dm/gfx/tools/dmsh_xlate_bin/dmsh_xlate.dll
#	dm/gfx/tools/dmsh_xlate_bin/dmsh_xlate.pdb
…d of a compare-and-swap loop

A task or mesh shader writes its primitive indices and count as bytes, which the
translator's shared memory byte store turned into a compare-and-swap loop on the
word. The 32 lanes of a workgroup write neighbouring bytes, so four lanes contend
for each word and the loop ran about four times a store, with a divergent retry
each time. Ocarina of Time's foliage mesh shaders have ten such stores a lane over
tens of thousands of workgroups a frame: 8.3-8.6 ms of GPU a frame on an M3 Pro,
a third of the frame, whatever the resolution.

GenerateSharedStore now clears the field with an atomic AND and sets it with an
atomic OR. Lanes writing other bytes of the word commute with both, so the word
ends as the loop left it, without retries. Two lanes writing the same byte at once
get a mix of their values where the loop gave one of the two; the hardware's order
between them is undefined as well. The helper also serves 16-bit stores and the
shared memory of compute shaders. A chain change (3938bdc7c7c0): translator
rebuilt, lock and header updated, the StageMemory test checks the new form.

DM_NVN_SKIP_MESH=1 drops every task / mesh draw, as DM_NVN_SKIP_FS drops draws by
fragment shader: how the cost was isolated.

Measured on an M3 Pro (18 GB), Kokiri Forest, the pass holding the foliage:
8.3-8.6 ms -> 3.5-4.0 ms a frame; the game's resolution controller then holds
1440x810 at 48 fps where it held 854x480 at 36-40 fps.
@Romen-Adama-Dev
Romen-Adama-Dev force-pushed the perf/shared-store-and-or branch from 4919040 to ebb6a39 Compare November 6, 2026 19:09
@Romen-Adama-Dev

Copy link
Copy Markdown
Contributor Author

Rebased on main e0dc79c (the chain changed under it with the HMMA commit): translator republished on the current sources, lock and header updated, chain 3938bdc7c7c0. The 64 StageMemory tests, the mesh stage, chain and docs tests pass again. Nothing else changed.

seedds added a commit that referenced this pull request Nov 7, 2026
…f a compare-and-swap loop on their word: task and mesh shaders' primitive indices no longer retry (chain f9f01e4f9994, PR #1503)

The translator stored a byte or half-word of shared memory (STS.U8 / .U16, and a mesh shader's primitive count and index bytes, 0x927 modes 0x88 and 0x80) through upstream Ryujinx's helper, a compare-and-swap loop on the word, as SPIR-V's shared memory is an array of words. A mesh workgroup's 32 threads write neighbouring index bytes, so four contend for each word and each store retries about four times. The helper is now named dmsh_shared_store8_<n> / dmsh_shared_store16_<n>, and dmsh_build.py shared_store_msl replaces its body in the Metal source with one store through a uchar pointer, or an aligned may_alias ushort one, over the threadgroup array (or the device array tg_spill makes of it). Metal's threadgroup memory takes byte stores, and each byte is its own memory location, so neighbours need no atomics; two threads storing the same byte leave one of the two values, as the loop and the console do. The pull request's atomic AND then OR needs no retries either, but leaves the OR of two values written to one byte, which the GPU test shows.

Pull request #1503 measured the AND / OR form on an M3 Pro in Kokiri Forest: the foliage pass from 8.3-8.6 ms to 3.5-4.0 ms, and the game's resolution from 854x480 to 1120x630. On an M6 (12 GPU cores), from one standing view on the forest deck, the loop, the store and AND / OR all measure 14.0-14.6 ms of GPU p50 (three runs each, one without the pipeline archive), against 10.3 ms with the mesh draws dropped: the store has not been measured on an M3. DM_NVN_SKIP_MESH=1 (the pull request's diagnostic) drops every task and mesh draw.

Checks: test_dmsh_shared_store.py (new: the rewrite, a spilled kernel, and on the GPU 64 threads storing neighbouring bytes and half-words over 64 dispatches; with DMSH_TEST_XLATE the translated index byte), test_dmsh_ampere_mem.py, test_dmsh_meshstage.py, test_nsp_chain.py, test_docs.py. Ocarina of Time's full build: 42,848 of 42,848 shaders, 515 with the store (500 mesh shaders, 15 compute kernels); its walk reaches Kokiri Forest with the title and the forest drawn. Mario Kart World's 18 mesh shaders with a byte store build for the Mac and iOS (not run). Regression: Ocarina of Time only, the one game with libraries on this chain here; the store gives what the loop gave.

Co-authored-by: Romen Adama Caetano Ramírez <56840314+Romen-Adama-Dev@users.noreply.github.com>
@seedds

seedds commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Thank you: landed in ffebfd5 (chain f9f01e4f9994), with you as co-author. Your diagnosis (the byte-store loop in the mesh shaders), your measurements and DM_NVN_SKIP_MESH are kept. The fix itself is different:

  • Instead of an atomic AND then OR, the shader build replaces the helper's loop with one store of the byte or half-word through a narrow pointer (dmsh_build.py shared_store_msl; the translator names the helper dmsh_shared_store8/16_<n> so the build can find it). Metal's threadgroup memory takes byte stores, and each byte is its own memory location, so neighbouring threads need no atomics.
  • The reason: with AND / OR, two threads writing the same byte leave the OR of both values, a value neither wrote; the new GPU test (test_dmsh_shared_store.py) shows exactly that for AND / OR. The store leaves one of the two values, as the loop and the console do.

Measured on an M6 (12 GPU cores), Ocarina of Time standing on the Kokiri Forest deck, three runs each: the loop, the store and your AND / OR all give 14.0-14.6 ms GPU p50, against 10.3 ms with DM_NVN_SKIP_MESH=1. So on this GPU the loop costs nothing measurable, and your M3 Pro gain could not be checked here. If you can, a run of main on your M3 Pro (after a shader rebuild for the new chain) would tell whether the store gives the same gain as your AND / OR; the reference entry is "Stage memory byte stores through a compare-and-swap loop" in docs/reference/shaders.md.

@seedds seedds closed this Nov 7, 2026
Aznracer pushed a commit to Aznracer/Mighty-Green-Kokiri-Kid that referenced this pull request Nov 8, 2026
…f a compare-and-swap loop on their word: task and mesh shaders' primitive indices no longer retry (chain f9f01e4f9994, PR seedds#1503)
Know-It-All-Brother pushed a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
…f a compare-and-swap loop on their word: task and mesh shaders' primitive indices no longer retry (chain f9f01e4f9994, PR seedds#1503)

The translator stored a byte or half-word of shared memory (STS.U8 / .U16, and a mesh shader's primitive count and index bytes, 0x927 modes 0x88 and 0x80) through upstream Ryujinx's helper, a compare-and-swap loop on the word, as SPIR-V's shared memory is an array of words. A mesh workgroup's 32 threads write neighbouring index bytes, so four contend for each word and each store retries about four times. The helper is now named dmsh_shared_store8_<n> / dmsh_shared_store16_<n>, and dmsh_build.py shared_store_msl replaces its body in the Metal source with one store through a uchar pointer, or an aligned may_alias ushort one, over the threadgroup array (or the device array tg_spill makes of it). Metal's threadgroup memory takes byte stores, and each byte is its own memory location, so neighbours need no atomics; two threads storing the same byte leave one of the two values, as the loop and the console do. The pull request's atomic AND then OR needs no retries either, but leaves the OR of two values written to one byte, which the GPU test shows.

Pull request seedds#1503 measured the AND / OR form on an M3 Pro in Kokiri Forest: the foliage pass from 8.3-8.6 ms to 3.5-4.0 ms, and the game's resolution from 854x480 to 1120x630. On an M6 (12 GPU cores), from one standing view on the forest deck, the loop, the store and AND / OR all measure 14.0-14.6 ms of GPU p50 (three runs each, one without the pipeline archive), against 10.3 ms with the mesh draws dropped: the store has not been measured on an M3. DM_NVN_SKIP_MESH=1 (the pull request's diagnostic) drops every task and mesh draw.

Checks: test_dmsh_shared_store.py (new: the rewrite, a spilled kernel, and on the GPU 64 threads storing neighbouring bytes and half-words over 64 dispatches; with DMSH_TEST_XLATE the translated index byte), test_dmsh_ampere_mem.py, test_dmsh_meshstage.py, test_nsp_chain.py, test_docs.py. Ocarina of Time's full build: 42,848 of 42,848 shaders, 515 with the store (500 mesh shaders, 15 compute kernels); its walk reaches Kokiri Forest with the title and the forest drawn. Mario Kart World's 18 mesh shaders with a byte store build for the Mac and iOS (not run). Regression: Ocarina of Time only, the one game with libraries on this chain here; the store gives what the loop gave.

Co-authored-by: Romen Adama Caetano Ramírez <56840314+Romen-Adama-Dev@users.noreply.github.com>
rafaeliga pushed a commit to rafaeliga/kid-with-sword-remake that referenced this pull request Nov 11, 2026
… keep their last pool range in private words (from the oot2 branch; chain 804433846e4b)

- oot2 5a1923b (PR seedds#1503): STS.U8 / .U16 and a task or mesh shader's stage-memory bytes (its primitive indices and
  count) were a compare-and-swap loop on their word; dmsh_build.py shared_store_msl makes each one byte or half-word
  store. Ocarina of Time's foliage mesh shaders (ten such stores a lane) took 8.3-8.6 ms of GPU a frame on an M3 Pro.
  Its nvn_geometry.inc part (background mesh pipeline builds) is not taken.
- oot2 6951110: the raw-read helper keeps the entry it chose last, with its range and delta, in seven private words,
  so a read inside that range needs no read of the pool table (Ocarina's skinned vertex shaders read 759 words a vertex
  from one pointer, about 7 ms of a frame in Hyrule Castle Town). It replaces the one-word hint and works with V21's
  vector loads; test_dmsh_raw_last's vector tests now pass the seven words.

Checked with DMSH_TEST_XLATE: test_dmsh_raw_last (4, two on the GPU), test_dmsh_shared_store, test_dmsh_float_add.

Co-authored-by: contributor <contributor@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0400486024DEC000 A game nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants