Skip to content

Preserve shared-memory writes during shader AST optimization - #1487

Open
thehobbit85 wants to merge 1 commit into
seedds:mainfrom
thehobbit85:codex/shader-optimizer-memory-effects
Open

thehobbit85 wants to merge 1 commit into
seedds:mainfrom
thehobbit85:codex/shader-optimizer-memory-effects

Conversation

@thehobbit85

@thehobbit85 thehobbit85 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1415 by preserving addressable shader call arguments and atomic writes whose result is discarded. The current main translator still reproduces both optimizer defects.

Current integration, 2026-11-10

Published commit ebf51ecb4c1524b2a51c0a062b4fb9d60a2342b9 rebases 72a05e090e2d7e02ba7745c459f0382dc56afb8d onto bbea4c90f136f478e25228bc3d4ab33c77420f2e. The optimizer patch and its assertions are unchanged. The rebuilt shader assembly combines them with main’s uniform-buffer bounds, vector loads, and other translator changes. It was published from source with .NET SDK 10.0.402. The main launcher, dependency graph, companion artifacts, and compiler pins are preserved. Combined chain b4168d6e0f71. The tool lock and apphost signature checks passed with Xcode 27.2 and Metal 32023.922.4 for both SDKs. AnyCPU metadata passed its fixture. Managed x64 and physical Intel execution remain open.

Validation on this revision

  • Both optimizer regression tests passed. The exact main bundle fails all four subcases by failing to translate the two subword stores and omitting the ADD and EXCH with discarded results. The four optimized candidate outputs each compiled and linked for macOS and iOS, for eight compiler checks.
  • The broader run recorded 232 passes, four failures and six errors in 242 tests. Exact main reproduces the same ten outcomes. Three Pokopia assertions still expect unguarded uniform reads, six mesh tests omit dmsh_fetch from their extracted fixture scope, and the ABI literal ratchet counts a third "?" string in nvn_mtl.m. The main bundle and the Pokopia and mesh fixture inputs are identical between 5e1c1aec and the final main pin. The ABI failure was also reproduced on exact final main.
  • The Super Mario Odyssey Mac host built. Main’s revised conditional-rendering fixtures passed in both encoder modes. Counter overlap passed 37 checks per mode, and render enable passed 52 checks per mode. The counter fixture checks draw predicates and retains the open GPU counter-byte ordering diagnostic. These targeted checks supersede the old Mario-profile fixture results for these cases. A full native suite on this revision remains open.
  • Pinned Python lint reports 339 findings against 336 baseline entries, with three excess findings in dmsh_build.py. Exact main reproduces the same diagnostics. The baseline and affected source are unchanged. Generated documentation, chain verification, and whitespace checks passed.

The impact scan identifies Breath of the Wild, Pokémon Scarlet, and Tears of the Kingdom through the shared NVN shader chain. Pokopia and Splatoon Raiders add shared-atomic coverage. Read-only discovery found no runnable Mac builds for these five under the inspected roots. Full game shader and gameplay comparisons and iOS-device validation remain open. Remote CI remains pending verification.

Historical published description, validation and original commit context

Fixes #1415 by preserving addressable shader call arguments and atomic writes whose result is discarded. Current main still reproduces both defects.

Current integration, 2026-11-09

Published commit 72a05e090e2d7e02ba7745c459f0382dc56afb8d rebases 1b92e34e9b4620cc7c7a11233a0382db5beddb97 onto 413e8808e0799eab8730688f082835aeeaca84c5. The optimizer patch and regression assertions are unchanged. The new main vector-load lowering, mesh scheduling, and shader builder changes are included. The combined shader assembly was rebuilt with .NET SDK 10.0.402. The universal apphost, runtime dependency graph, and compiler pins remain those from main. The chain is 3ff28178f8d0.

Validation on this revision

  • 207 focused tests passed with zero failures, errors, or skips. The run covers Ampere memory and ALU, shared stores, vector global loads, extended multiply, mesh stages, HMMA, handle tables, docs, chain and AnyCPU metadata. With the exact main translator, the same focused regression assertions reproduced four failures. Two subword stores failed in SPIR-V call generation, and the discarded-result ADD and EXCH were absent from generated Metal. All four cases passed with the candidate.
  • The four optimized shaders each compiled and linked on macOS and iOS Metal. These eight compiler checks are separate from the synthetic shared-store GPU fixture. The pinned tool lock and apphost signature checks passed with Xcode 27.2 and Metal 32023.922.4. Managed x64 execution and physical Intel validation remain open.
  • The Mac host built for Super Mario Odyssey. The full graphics command exited 2 at t_render_counter_overlap after the four repaired presentation-crop modes passed 112 checks each. The follow-up targets also exposed nine failed assertions in t_render_enable_queue. Exact main reproduces both failures with the same Mario profile, which sets DM_GAME_RENDER_COND=0. Under that profile the host deliberately draws through disabled predicates. The failing counter case's GPU bytes matched the expected zero. Both fixture recipes stopped before their second mode on this profile.
  • With the conditional-rendering profile for Tears of the Kingdom, exact main passes the counter pixel assertion in both modes. All 42 checks pass without the encoder thread. Threaded mode still fails two counter-byte assertions, and the existing high-word diagnostic remains open. These are synthetic fixtures. The other follow-up targets passed, including the twelve unreached after the first follow-up. Six optional Wonder reflection samples were unavailable. The full graphics suite is incomplete.
  • Pinned Python lint reports 339 findings against 336 baseline entries, with three excess findings in dm/tools/unity_bucket.py. Exact current main produces the same diagnostic log. The lint baseline is unchanged. Generated documentation and whitespace checks passed.

The impact scan names Breath of the Wild, Pokémon Scarlet, and Tears of the Kingdom through the shared NVN shader chain. Pokopia and Splatoon Raiders are additional comparison targets for shared atomics. Read-only game discovery found no runnable Mac builds for these five under the inspected game roots. Full game shader and gameplay comparisons and iOS-device validation remain open. Remote CI on this head is pending verification.

Historical published description and original validation

Fixes #1415. Preserve addressable shader call arguments and atomic writes whose return values are discarded. Main still lacks the optimizer changes; its translator source and binary are unchanged from the November 7 failing control. The original optimizer patch, assertions and commit message remain intact.

Current validation — November 8

Rebased the confirmed generated-header conflict onto main 555122a9069235441d5a88955105397ec6d53686; tested head 1b92e34e9b4620cc7c7a11233a0382db5beddb97, chain 07e6ae1e4d33. The combined source retains main's shader-output masks, handle-table correction and pipeline build changes. All five optimizer source changes, the translator binaries and lock fingerprints are byte-identical to the previous reviewed PR head. Only the generated chain header changes in the feature range-diff; all 87 upstream-only paths match main.

Fresh validation on Apple M2 Max, Xcode 27.2 and supported Metal 32023.922.4:

  • DMSH_TEST_XLATE=$PWD/dm/gfx/tools/dmsh_xlate_bin/dmsh_xlate /opt/homebrew/bin/python3 -B -m unittest dm.gfx.tests.test_dmsh_ampere_mem dm.gfx.tests.test_dmsh_ampere_alu dm.gfx.tests.test_dmsh_shared_store dm.tools.tests.test_docs dm.tools.tests.test_nsp_chain: 143 passed, no skips. This includes optimized store/atomic regression cases and the native shared-byte/halfword Metal compilation and GPU fixture.
  • /opt/homebrew/bin/python3 -B -m unittest dm.tools.tests.test_dmsh_pool_arrays: seven passed, covering main's updated handle-table behavior. tool_lock.py --check-tools passed with the unchanged supported pins.
  • make -C dm/host GAME=010015100B514000 -j3 passed. make -C dm/gfx/tests GAME=010015100B514000 -j3 run did not complete: it stopped with SIGSEGV in t_present_crop. The crash is at nvn_queue_present (nvn_queue.inc:207), where current main reads q->serial; the fixture calls present(NULL, ...) at t_present_crop.m:197. The fixture, runtime and build recipes match exact main; only the generated chain string differs. A separate build from an exact 555122a9 source export reproduced the same SIGSEGV under the normal recipe. With the optional Metal debug layer enabled, that main build instead stops earlier at Apple MetalFX’s _outputTextureBarrierStages assertion; these are distinct failures. Earlier executed graphics cases passed, but later cases were not reached. No assertion, runtime guard or baseline was weakened.
  • Pinned Ruff 0.17.2 / Pyright 1.1.414 does not pass: 376 findings against the 371 baseline, five new and zero stale. Exporting all exact-main Python/type/configuration inputs reproduces the same diagnostics byte for byte in dm/tools/tests/test_patches.py, from main's new frame-step matching tests. No baseline growth, suppression or feature-branch workaround was added. This shared failure remains unresolved.

The previous eight optimized/unoptimized synthetic SPIR-V/Metal readbacks and four additional native-store Mac/iOS compiler/readback cases remain historical on e284a46b; translator/source/lock identity was verified before retaining that evidence. They were not repeated or relabeled as new game validation.

The fresh impact scan identifies Breath of the Wild, Pokémon Scarlet and Tears of the Kingdom through the general shader rule; Pokopia and Splatoon Raiders also exercise shared atomics. The required scripts/regress.py check --games 01007EF00011E000,0100A3D008C5C000,0100F2C0115B6000,0400A3A01EE4C000,0400380020056000 stopped with exit 2 because no qualifying local Mac builds of those five are available. No gameplay, full game-shader rebuild or iOS device behavior is claimed. Remote CI on this replacement head remains pending verification.

Earlier implementation, validation and preserved commit context (historical)

Fixes #1415. Shader optimization must preserve addressable call arguments and atomic writes even when their return values are discarded. These defects remain in main: its exact shipped translator fails both optimized subword-store cases and both discarded-result atomic cases. The original five-file optimizer patch and its assertions are retained.

Current validation — November 7

Tested e284a46b1b791551a7c02550fdcd308ed0e6e22e against main 01abf27977aa4a7366c4a14bc58d9eae75f1d8e6 on Apple M2 Max with Xcode 27.2 and supported Metal 32023.922.4. Rebuilt Ryujinx.Graphics.Shader.dll from the combined sources, updated only that lock fingerprint, and regenerated chain 2fd0305821c0. Main's other tools and compiler pins remain intact. The original commit message is preserved; range-diff and remaining-diff review passed.

  • DMSH_TEST_XLATE=$PWD/dm/gfx/tools/dmsh_xlate_bin/dmsh_xlate /opt/homebrew/bin/python3 -B -m unittest dm.gfx.tests.test_dmsh_ampere_mem dm.gfx.tests.test_dmsh_ampere_alu dm.gfx.tests.test_dmsh_shared_store dm.tools.tests.test_docs dm.tools.tests.test_nsp_chain: 143 tests passed. Pinned Ruff/Pyright: 371 baseline findings, zero new and zero stale.
  • All eight optimized/unoptimized synthetic STS.U8, STS.U16, ATOMS.ADD and ATOMS.EXCH shaders passed SPIR-V validation and Mac GPU readback. The four unoptimized binaries exactly match the current main translator's output. Observed words were 0x25223344, 0x00253344, 0x11223349 and 0x00000005, respectively.
  • Main's newer native-byte/halfword Metal rewrite was also applied to all four translated subword cases. Each passed GPU readback and both Mac/iOS compiler checks. This verifies compatibility with the upstream rewrite without altering the existing raw-translator assertions.
  • make -C dm/host GAME=010015100B514000 -j3 and make -C dm/gfx/tests GAME=010015100B514000 -j3 run completed successfully. All executed cases passed; six optional Wonder reflection samples were unavailable, and two texture-upload poll cases were inapplicable to the selected profile. The former native-link failure is historical. Existing compiler warnings remain in unchanged main source; no warnings were suppressed.

The impact scan identifies Breath of the Wild, Pokémon Scarlet and Tears of the Kingdom under the general shader rule; Pokopia and Splatoon Raiders also use shared atomics. scripts/regress.py check --games for those five stopped at inventory because none has a qualifying local Mac build. No game execution or full game-shader rebuild is claimed. Synthetic iOS compilation does not establish device behavior. Remote CI on this replacement head awaits verification.

Earlier implementation and validation (historical revisions)

Fixes #1415. With shader AST optimization enabled, 8- and 16-bit shared stores need an addressable argument for the write helper. Atomic add and exchange must also execute when the shader discards their return value. This change preserves both properties and includes main’s Ampere global compare-and-swap support.

Current validation

Tested 28e2564e against main b2d46b86 on an Apple M2 Max. Built the shader assembly from the combined C# sources and regenerated the tool lock and shader-chain header (5264d54bea94). The supported compiler fingerprints matched. The exact main translator failed all four optimized regression cases. The rebuilt translator passed the focused Ampere memory, arithmetic, docs and chain suite (111 tests), and pinned Ruff/Pyright reported 371 baseline findings with zero new or stale findings.

All eight synthetic shaders passed SPIR-V validation and Metal GPU readback. With optimization on and off, the observed shared word was 0x25223344 for STS.U8, 0x00253344 for STS.U16, 0x11223349 for ATOMS.ADD, and 0x00000005 for ATOMS.EXCH. The Mac host built successfully. The full native graphics suite stopped while linking t_gfx_suspend because its target omits the dm_vi_frame_queued stub. The target, fixture, caller and stub sources match the tested main revision. Earlier full-suite and Python results are recorded below.

The impact scan identified Breath of the Wild, Pokémon Scarlet and Tears of the Kingdom through the general shader rule. Pokopia and Splatoon Raiders also use shared atomics. The regression harness found no Mac build of these five games. This revision has synthetic shader and Mac GPU coverage. Gameplay, full game-shader builds and iOS behavior remain unverified.

Earlier implementation and validation

Fixes #1415.

With debug: false, the shader AST optimizer could substitute a value for a local whose address the SPIR-V backend passes to a function. That broke the write helper used by 8- and 16-bit shared stores. It could also discard an atomic when the shader sent its return value to RZ, losing the write to shared memory.

The optimizer now preserves those argument locals and keeps calls and atomic operations at their original execution point. The structured and lower-level optimizers share the same side-effect check. This includes the rebuilt translator assembly, its lock entry and shader chain 2070b9d404dc.

Validation

Tested on an Apple M2 Max running macOS 26.6 with Xcode 27.2, against main 0f510fd5.

  • Ran the two new Shared.test_optimized_* regressions against main's shipped translator. Both store widths failed translation and both discarded-result atomics disappeared. All four cases passed with the rebuilt translator. The full dm/gfx/tests/test_dmsh_ampere_mem.py module passed all 66 tests with DMSH_TEST_XLATE set.
  • Translated synthetic STS.U8, STS.U16, ATOMS.ADD and ATOMS.EXCH jobs with each debug setting. All eight modules passed spirv-val --target-env vulkan1.1. For GPU readback, the test harness added initialization and observation of one shared word around the generated Metal body. Starting from 0x11223344, both settings produced 0x25223344, 0x00253344, 0x11223349 and 0x00000005, respectively.
  • Built Ryujinx.Graphics.Shader.csproj in Release with .NET SDK 10.0.402, then verified the shipped assembly and tool lock. make -C dm/gfx/tests GAME=010015100B514000 -j3 run passed locally. The pinned Ruff and Pyright checks reported zero new findings.
  • The CI Python module list ran 1,718 tests with 13 skips and one failure. The same texture-pool expectation, Reflect.test_table_of_uniform_handles, also fails on unmodified main. It expects a generated texture lookup without the existing dm_pool_mark wrapper.

Game coverage

The production shader build uses XLATE_DEBUG=1 by default. The reported defect requires the alternate optimization mode. regress.py impact listed Breath of the Wild, Pokémon Scarlet and Tears of the Kingdom under the broad shader rule. Pokopia and Splatoon Raiders also use shared atomics. This pass used synthetic shaders only. Game builds, gameplay and iOS were untested.

Preserved commit message
Preserve shader call arguments and atomic writes during AST optimization

Keep SPIR-V input argument storage addressable and keep effectful expressions at their definition. Cover optimized shared subword stores and discarded-result atomic add and exchange.

Fixes #1415

@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders labels Nov 6, 2026
javierdv7 pushed a commit to javierdv7/nxnative that referenced this pull request Nov 6, 2026
…#1462's CAS, seedds#1487, seedds#1329, seedds#1410, seedds#1411, seedds#1412, seedds#1357): lock and chain header; conflict leftovers removed from shaders.md and test_dmsh_ampere_alu.py
@thehobbit85
thehobbit85 force-pushed the codex/shader-optimizer-memory-effects branch 4 times, most recently from 1b92e34 to 72a05e0 Compare November 9, 2026 02:38
Keep SPIR-V input argument storage addressable and keep effectful expressions at their definition. Cover optimized shared subword stores and discarded-result atomic add and exchange.

Fixes seedds#1415
@thehobbit85
thehobbit85 force-pushed the codex/shader-optimizer-memory-effects branch from 72a05e0 to ebf51ec Compare November 10, 2026 02:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders

Projects

None yet

Development

Successfully merging this pull request may close these issues.

With debug: false, an 8 / 16-bit STS throws and an ATOMS to RZ is dropped

2 participants