Skip to content

Working fallback mesh shaders (AMD RDNA1) and various performance and accuracy fixes - #1023

Closed
rakefire-impala wants to merge 18 commits into
NotRexed:mainfrom
rakefire-impala:working_fallback_mesh_shaders_RDNA1
Closed

rakefire-impala wants to merge 18 commits into
NotRexed:mainfrom
rakefire-impala:working_fallback_mesh_shaders_RDNA1

Conversation

@rakefire-impala

@rakefire-impala rakefire-impala commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Mesh shader fallback for GPUs without mesh shaders (RDNA1 and older), and fixes for AMD

This brings back #640 and makes it work. On an AMD RX 5500M (RDNA1, no VK_EXT_mesh_shader), the foliage, the floor grass, the ivy and the door of Link's house now draw. It also fixes what is likely the cause of the RTX 4060 Ti device loss that got #640 reverted (806c566), two AMD rendering bugs, and several texture problems on low-VRAM cards.

The full write-up, covering what was broken, how it was fixed and how it behaves, is in docs/mesh_fallback_rdna1.md.

Important

Updated since the first version (10 new commits, 9-18 below):

  • Pausing and resuming after alt-tab is fast again. While the game was paused or in the background, a texture budget that kept shrinking evicted about 1,000 textures (2,851 MB down to 1,137 MB) in a few seconds, and unpausing had to make them all again. That no longer happens.
  • No more black glyphs instead of the hearts, and no broken minimap view cone on 4 GB cards. Same cause as above.
  • The "radioactive" glowing stems and flowers are fixed. This was a shader translator bug on every GPU, not the fallback. Upstream fixed it in PR Fix glowing foliage stems and branches (register marker collisions) #759, which is cherry-picked here.
  • The black stepped patch on the Deku Tree's floor is gone, and walking stutters less, from two upstream v2.0.6 AMD texture fixes, cherry-picked.
  • New debugging tools: a live switch file to hide, list, read back or inspect fallback draws, with no restart.

Screenshots

Both are from an AMD RX 5500M laptop (4 GB of VRAM, no mesh shaders), with the fallback on Auto, 50% resolution, the handheld profile and frame generation 2x.

Kokiri Forest: the foliage, ground cover, trees and textures all draw, and the minimap is correct.

Image

House: the floor grass, the ivy, the door and the lighting are correct. Without the fallback, the floor here is flat moss with no grass and the door does not render. On this card, before the AMD fixes, the lighting came out dark and a basket glowed white.

Image

Status

Part Status
Foliage, floor grass, ivy, vines, the door of Link's house Working on the test laptop
Lighting and bloom on AMD Working on the test laptop
GPUs with mesh shaders (the #640 device loss) Fixed in the code, not tested on such a GPU
Texture noise after alt-tab on low-VRAM cards Working, but further testing needed
New: slow resume after pause or alt-tab, black hearts, broken minimap cone (4 GB cards) Working, but further testing needed
New: glowing stems and flowers in Kokiri Forest Fixed by upstream PR #759 (a translator bug on every GPU)
New: black patch on the Deku Tree's floor, terrain upload stutters on AMD Fixed by upstream v2.0.6 commits

Tested on: a low-VRAM laptop

All of this was tested on one laptop: an AMD Radeon RX 5500M (RDNA1) with 4 GB of VRAM, on Windows 11 with AMD's driver. Vulkan lets Hookshot use only about 3.1 to 3.4 GB of that, which is well below the 8 GB the readme recommends. Several of the bugs below only showed up because of the low VRAM: the lock-up, the white distant foliage, the texture noise, the slow resume and the black hearts. The Minimum (4GB) preset is the one to use on a card like this.

These frame rates come from the logs. They are not a controlled benchmark.

State Frame rate GPU time per frame
Docked settings, before the grass fix (Link's house) ~15 fps ~54 ms
Minimum (4GB) preset (Link's house) ~19 fps ~41 ms
Minimum preset, after the fallback's later fixes (Link's house) ~34-38 fps ~17 ms
New: after the v2.0.6 texture fixes (inside the Deku Tree, so not the same spot) ~44 fps not measured

How it behaves

  • It is picked at every start. If the GPU has mesh shaders, the game draws as before and nothing of the fallback is built or cached. If it has none, the fallback turns on by itself, provided the driver has buffer device addresses (Vulkan 1.2).
  • There is a toggle: Insert menu → Graphics → Mesh shader fallback.
    • Auto (the default) uses the fallback only on GPUs without mesh shaders.
    • Off never uses it.
    • On uses it on every GPU and does not turn on the GPU's own mesh shaders.
    • A change applies after a restart. The setting is not part of the Quality preset, so changing it keeps the preset's VRAM caps. RX_VK_NO_MESH_FALLBACK=1 still works and has the same effect as Off.
  • On is for AMD cards that have mesh shaders but hang with them. Lichii's #814 found that native task and mesh draws hang AMD RX 6000 and newer cards. On Windows that shows as a driver timeout (TDR), and on Linux as GPU resets with RADV. Lichii therefore uses its fallback by default on AMD. This PR keeps Auto as the default, and players with those cards can switch to On. The hang is Lichii's finding and has not been confirmed in Hookshot yet.
  • Every fallback program is compiled at startup, so nothing pops in white during play. The third "Preparing shaders" count, about 11,000 on the test laptop, now includes about 1,350 fallback programs. AMD can take 10 to 12 seconds to compile one, so the first start after a translator change or a driver update takes a few minutes. Later starts load them from the driver's cache in seconds.
  • The launch warning now only appears when it's true: on a GPU with no mesh shaders where the fallback can't run either. It explains why: either the fallback was turned off, and it says how to turn it back on, or the driver lacks buffer device addresses. It no longer says Hookshot needs an RTX 20 / RX 6000 / Arc card.

What was broken, and the fixes

#640's shader translation

These were found with a new offline validator, scripts/validate_spirv.py. At first, almost every one of the ~1,350 fallback programs failed spirv-val. Now all 1,347 pass.

  • Payload address: the payload's address was read from the push constants using its byte offset (64) as a member index, so every task-driven program read and wrote its payload at a garbage address.
  • Missing Aligned: device-address loads and stores had no Aligned, which Vulkan requires.
  • Partial payload copy: each task thread copied only 1 word in 32 of its payload, because the copy loop stepped by thread.
  • Wrong mesh workgroup index: a mesh workgroup's index within its task was its global slot (up to 13,855) instead of 0–31, so it picked a meshlet past the end of its task's list.
  • Invalid SPIR-V: a loop counter was declared as a Function variable outside the first block, and a bool was stored into a u32.

#640's draw side

  • The freeze: a compute pipeline the driver refused was rebuilt on every draw, and the stages built before it leaked. AMD also took seconds to compile each pipeline inside the draw. Pipelines now compile on a worker thread, and the draw is skipped until they are ready.
  • The lock-up, and the white distant foliage: every program had its own buffers on every recording thread, about 2.2 GB for 95 programs. When VRAM ran out, new buffers could not be made, and those programs stopped drawing for good. Foliage that lost its colour pass but kept its depth pass came out white. All of a thread's fallback draws now share one 72 MB pair of buffers, with a barrier between draws.
  • Missing floor grass, part 1: a draw had only 1,024 slots, each sized for its program's maximum, so about 93% of the grass was dropped. There are now 16,384 slots. A mesh workgroup claims records only for the triangles it outputs, and the draw is indirect, sized from what was claimed. That also removed the GPU's work on empty vertices.
  • Missing floor grass, part 2: the grass is drawn with DrawMeshTasksIndirect, where the task count lives in GPU memory. The mesh stage was launched with the CPU's copy of that count, which is 0, so no mesh workgroup ever ran for indirect task draws. Task threads now count the mesh work on the GPU, and the mesh stage is launched from that count.
  • Why the door and ivy appeared before the grass: they started drawing once the payload and index fixes above were in, but the grass is an indirect draw, so it needed the fix just above as well.
  • The door was white at spawn while its colour pass waited for slow compiles. The fix is the startup precompile described above.
  • New, hardening: the generated vertex stage now starts every output at zero, as the console's stage buffer does, instead of leaving unwritten components undefined. Fragment-stage butterfly shuffles now take the same AMD workaround as the other shuffles. Neither was the cause of the glowing foliage below.

GPUs with mesh shaders: the #640 regression

  • [AMD RX 5500 XT / RDNA1] Missing Geometry Due to Unsupported Mesh Shaders — Lichii Compute Shader Fallback Works #640 changed build_graphics (which builds every graphics pipeline) to drop all TASK and MESH stages for every program. Fallback stages are compute stages, so in practice this only ever hit native mesh programs: on a GPU with mesh shaders, every native mesh pipeline was built with only a fragment stage. That is very likely the RTX 4060 Ti device loss, which happened with the fallback off.
  • build_graphics now skips only a fallback program's compute stages.
  • The push-constant range is back to main's size when the fallback is off.
  • Vertex input is read from stage 0 again.
  • Native task and mesh shader code is byte for byte main's.

AMD lighting and bloom (not specific to the fallback)

  • 64-lane graphics warps: AMD only allows compute shaders to be pinned to 32 lanes, so fragment shaders run 64 lanes wide while the translator assumed 32. Graphics stages on such GPUs now use the translator's 64-lane path, and compute keeps the old one. Giving compute the 64-lane path too made shader preparation take hours. The program-cache stamp now includes these settings.
  • Lichii #273: fragment-shader shuffles on AMD's proprietary driver now go through the broadcast workaround (a basket glowed white). Depth-compare textures are never treated as integer images.

Texture noise after alt-tab (working, but further testing needed)

This one is a bug in Hookshot's own texture code, unrelated to the fallback:

  1. With the game in the background, about a thousand textures got evicted.
  2. Re-making them requires the BC3 re-encode scratch, whose output buffer could only live in VRAM, and VRAM was full.
  3. Every re-make failed, but each texture was marked as uploaded anyway, so the empty BC3 images were drawn as noise and never retried.

Now the buffer falls back to system memory, and a failed BC3 upload is retried 30 frames later. Each failure is logged as vk: BC3 texture ... its upload failed (...). This was only tested briefly.

New: slow resume after pause or alt-tab, black hearts, broken minimap cone (working, further testing needed)

Pausing and resuming, including after the game loses focus, is now fast. Before, unpausing could hang for a long time on a 4 GB card.

The cause was a texture budget that ratcheted down:

  1. Every 10 frames, when the driver said Hookshot was above 97% of its allowance, the texture budget was set below what was held, and textures unused for 60 frames (about a second) were evicted.
  2. Evicting frees space inside the memory blocks the allocator already holds, but the driver still counts those blocks. So its count stayed at the limit (3,337 of 3,337 MB, with 1,224 MB free inside the blocks), and 10 frames later the budget shrank again.
  3. While paused, nothing in the world is drawn, so its textures drained from 2,851 MB to 1,137 MB in a few seconds. Unpausing had to make about 1,000 of them again: ASTC decode, BC3 re-encode and upload.

The same squeeze broke the HUD. The hearts and the minimap's view cone are small textures the GPU draws once and the game samples again about 60 frames later. Their pixels exist only in the GPU image. Once evicted, they came back over memory that never got those pixels, as black glyphs, until an area change made the game redraw them.

Fix: the 97% squeeze no longer runs while the allocator's blocks hold at least an eighth of the allowance free. New textures go into that space without the driver's count growing. The other rule, which judges pressure after taking that free space off, still evicts when memory is really short. The VRAM log line says not squeezed when this applies, and RX_VK_SQUEEZE_ALWAYS=1 restores the old behaviour. The log also now names each small GPU-drawn texture evicted with its pixels lost.

New: glowing "radioactive" stems and flowers (fixed by upstream PR #759)

In Kokiri Forest, stems glowed cyan, petals bloomed white and bushes had coloured streaks. Upstream had the same report on GPUs with mesh shaders (#757 on an RTX 5080, #874, #986), so this was not the fallback.

  • How it was found: the new readback tool showed the geometry was right. The depth prepass and the colour pass had bit-identical positions and triangles in the same frame, with no NaNs. The stems' colour pass computes light-probe (spherical-harmonics) lighting per vertex, and about a third of the stems came out at hundreds to thousands, some negative.
  • The cause: the translator renames high registers through temporary markers, and a marker could take the number of an operand the same FFMA still read, so the math used the wrong value.
  • The fix: upstream PR Fix glowing foliage stems and branches (register marker collisions) #759 (deaf4ff), cherry-picked with its author kept. Its regression test fails on this branch's base without the fix and passes with it. In game, the stems and flowers look right.

New: black patch on the Deku Tree's floor, and terrain stutters on AMD (fixed by upstream v2.0.6 commits)

Inside the Deku Tree, a large stepped region of the floor sometimes turned black. It stayed with every fallback draw hidden, so the fallback doesn't draw it. This branch still had two AMD texture paths that upstream fixed in v2.0.6 (tracking issue #1006):

  • The uneven-BC3 resize (vk: BC3 levels that halve evenly on AMD (the terrain grid in Hyrule Field) #705) re-encoded the terrain's 1365×1365×90 material arrays at 1536×1536. Upstream measured 7× the GPU time on an RX 6700 XT with it. Here each of those 57 MB arrays took 80-93 ms to upload, a stutter every time.
  • The BC3 re-encode of ASTC arrays, several layers per run, left dark blocks on terrain on AMD.

Both upstream commits are cherry-picked: dc2f4dc (the resize is opt-in again, RX_VK_UNEVEN_BC=0 turns it on) and ec89f92 (#956, one layer per run). In game, the black patch did not come back, walking is smoother, the frame rate rose (table above), and VRAM is no longer pinned at the limit. The thin grid lines at terrain seams that upstream names as the trade-off were not seen.

New: debugging tools

A live switch file, hookshot_fallback_debug.txt in the game folder, is read again within half a second of a change, with no restart and no shader preparation:

  • off hides every fallback draw, and a key's hex prefix hides that program.
  • list logs each fallback program drawn.
  • read <prefix> copies a program's fallback records back to the CPU, all in one frame, then logs each value's range, NaNs and infinities, and saves them to a .bin file.
  • info <prefix> logs a draw's render targets and formats, write masks, blending and depth test.

RX_VK_WARP64=0|1 and RX_VK_FRAG_SHUFFLE_FALLBACK=0|1 force the AMD warp settings for A/B tests.

Commits

  1. 5918e80 Re-apply PR [AMD RX 5500 XT / RDNA1] Missing Geometry Due to Unsupported Mesh Shaders — Lichii Compute Shader Fallback Works #640 (undoes 806c566 and 449382e)
  2. dc61d57 tools: offline SPIR-V validator (scripts/validate_spirv.py)
  3. b17e373 AMD: 64-lane graphics warps, Lichii AMD RX6600 , I5-11400f , 16gb ram Crash after intro #273
  4. 6b5e629 vk: GPUs with mesh shaders draw as main again ([AMD RX 5500 XT / RDNA1] Missing Geometry Due to Unsupported Mesh Shaders — Lichii Compute Shader Fallback Works #640's broken native mesh pipelines)
  5. cb0d3ef menu: Mesh shader fallback (Auto / Off / On), and the launch warning text
  6. 4e36b5d mesh fallback: shader translation, draw side, memory, floor grass, startup precompile
  7. 353bd4a textures: retry failed BC3 uploads, with a system-memory fallback for the scratch buffer (working, further testing needed)
  8. 214096b docs: docs/mesh_fallback_rdna1.md and readme updates

New in this update:

  1. 5c2004f mesh fallback: zeroed vertex-stage outputs, butterfly shuffles through the AMD workaround, the live debug switch
  2. 55b7ee5 docs: glowing stems and flowers, as a known issue at the time
  3. 9218be7 mesh fallback debug: read (records readback) and info (draw state) in the live switch
  4. 46e8c3a cherry-pick of upstream PR Fix glowing foliage stems and branches (register marker collisions) #759 (deaf4ff): register markers never alias an original operand (glowing stems)
  5. f8c00a4 docs: the cause of the glowing stems and flowers (item 11)
  6. 5ff934f textures: no driver's-limit squeeze while the allocator's blocks hold room (fast resume, HUD fixed); log lost GPU-drawn textures
  7. 2709765 docs: the squeeze fix (item 12)
  8. a35cbf7 cherry-pick of upstream ec89f92 (BC3 re-encode of ASTC arrays: one layer a run (Hyrule Field grid lines) #956): BC3 re-encode of ASTC arrays one layer per run
  9. a34f6f6 cherry-pick of upstream dc2f4dc: the AMD uneven-BC3 resize (vk: BC3 levels that halve evenly on AMD (the terrain grid in Hyrule Field) #705) opt-in again
  10. 16229ef docs: the Deku Tree black patch and terrain stutters (item 13)

Testing

  • Played on the RX 5500M:
    • the foliage, floor grass, ivy, vines and door draw, the lighting and bloom are correct, there's no lock-up, and alt-tabbing no longer leaves noise
    • new: pausing and resuming, including after alt-tab, is quick; the hearts and the minimap cone stay correct
    • new: the Kokiri stems and flowers no longer glow; the Deku Tree's floor patch did not come back; walking stutters less
  • spirv-val: 1,347 fallback programs, 0 failures (run with --uniform-buffer-standard-layout --scalar-block-layout).
  • Tests:
  • Per commit: each commit's tests were run against that commit's own state. Only the final states were compiled and played. test_fps_overlay_order fails the same way on main: its fixture is missing <initializer_list>.

Needs testing / review notes

  • A GPU with mesh shaders, ideally an RTX card: please confirm the device loss is gone and that nothing else changed.
  • Other GPUs without mesh shaders (RX 5000, RX 400/500, GTX 10 series) are untested.
  • AMD RX 6000 or newer: try Mesh shader fallback set to On if they hang.
  • The texture fixes need more play time, and testing on other low-VRAM cards. With the squeeze fix the driver may page VRAM to system memory instead of Hookshot evicting; that wasn't seen here, but it's the risk.
  • Fix glowing foliage stems and branches (register marker collisions) #759 on AMD: upstream has an open report of green diagonal stripes on an RX 7600 that may come from Fix glowing foliage stems and branches (register marker collisions) #759 (its A/B switch, RX_VK_MARKERS_V205=1, is on main, not on this branch). None were seen on the RX 5500M.
  • Rebase: the branch is based on 232eecb, and main has moved on since, so it will need a rebase. Commits 12, 16 and 17 are cherry-picks of commits already on main, so a rebase should drop them as duplicates.

Known limits

  • Draw order: semi-transparent mesh geometry can occasionally draw out of order, because records land in the order the GPU claims them. Opaque foliage isn't affected. Lichii's fallback has the same limit.
  • Only one GPU tested so far.
  • One big hitch remains on 4 GB cards: the 12288×12288 shadow map (288 MB) was once made again and uploaded whole, taking about 2 seconds, probably on an area change. Other first-use texture uploads can still take up to ~100 ms.

Credits

#640 (shayder) for the fallback this builds on. Lichii (zortos293 and contributors) for the fallback design (#569), the AMD fixes (#273), the AMD hang finding (#814), and their notes on how the game draws its foliage. Upstream PR #759 for the register-marker fix, YunaraDebugs for #956, and NotRexed for making the AMD resize opt-in. The reporters of #757, #874, #986 and #1006, whose reports pointed to the upstream fixes.

rakefire-impala and others added 18 commits November 8, 2026 20:18
…ut mesh shaders

Undoes 806c566 (the revert of ea4eced) and 449382e (the prediction reader's
change for that revert): the program cache records carry the mesh-as-compute
fields again, so program_cache_each skips them as it did before.

For testing on a Radeon RX 5500M (no VK_EXT_mesh_shader): missing foliage
and the door at spawn. The revert's device loss was on a GPU with mesh
shaders (fallback off); RX_VK_NO_MESH_FALLBACK=1 turns the fallback off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rv.py)

hookshot_programs.bin keeps every translated program's SPIR-V. The script walks its records
(by default the mesh-as-compute ones, kind 4; --all or --key for others), runs each stage
through the Vulkan SDK's spirv-val, and checks what spirv-val does not know: a workgroup's
invocations and its workgroup memory against the GPU's limits (RX 5500M: 1024, 64 KB).
A failing stage is written out as .spv, .spvasm (spirv-dis) and .txt with the reasons, and
the summary groups the validator's errors by message.

It found every SPIR-V bug of the mesh fallback fixed on this branch: the payload address
read by byte offset instead of member index, PhysicalStorageBuffer accesses without
Aligned, a Function variable outside the first block, a bool stored into a u32. After the
fixes: 1347 mesh-as-compute programs, 0 stages failed (with --uniform-buffer-standard-layout
--scalar-block-layout, as Lichii validates).

tests/test_validate_spirv.py builds a small cache with a valid module over the workgroup
memory limit and an invalid one, and checks both are found.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…les and depth-compare textures

Fixes the dark lighting and the white-glowing objects (a basket in Link's house) on an
AMD RX 5500M.

- Warp size: AMD GPUs let only compute pin its subgroups to NVN's 32 lanes
  (requiredSubgroupSizeStages 0x20 on the RX 5500M); a fragment or vertex stage runs at
  64. The translator was told 32 lanes everywhere (warp_size_potentially_larger_than_guest
  false), so the ballots and shuffles of the game's fragment shaders ran across both halves
  of a 64-lane wave. make_profile now sets it, and max_subgroup_size to the device's most,
  when graphics stages cannot be pinned and the default is wider than 32: Eden's long path
  keeps each 32-lane guest warp to its half, and is right at 32 lanes too. NVIDIA (32
  lanes) is unchanged.
- Compute and the mesh fallback's compute stages keep the short path (profile_pinned):
  they are pinned to 32. Giving every compute program the long path made its SPIR-V new and
  longer, missing the driver's cache, and AMD compiled shaders for hours at start.
- Lichii NotRexed#273: on AMD's proprietary driver, fragment-stage subgroup shuffles take the
  broadcast fallback (has_broken_spirv_fragment_shuffle); and a depth-compare texture is
  never an integer image, whatever its descriptor says (the lighting's shadow samples).
- The device's driver ID and subgroup sizes are read at start; the program cache's stamp
  includes the profile's warp choices, so a cache translated for another GPU's warps is
  not used.
- The mesh stage is told its task program's threads (RuntimeInfo::mesh_fallback_task_threads),
  used by the mesh fallback's fixes that follow.

tests/test_amd_fragment_warp.py checks the profile, the pinned profile's uses, the stamp,
the shuffle fallback and the depth-compare rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e mesh pipelines had no task or mesh stage)

NotRexed#640 was reverted upstream (806c566) after an RTX 4060 Ti, a GPU WITH mesh shaders and so
with the fallback off, lost the device. Found here: NotRexed#640 made build_graphics, which builds
every graphics pipeline (in play, on a worker, as pipeline libraries, in the precompile and
the prediction), leave out every stage whose flags had COMPUTE, TASK or MESH. Fallback stages
are already VK_SHADER_STAGE_COMPUTE_BIT, so that only ever matched native task and mesh
stages: on a GPU with mesh shaders every native mesh pipeline was made with nothing before
its fragment stage.

- build_graphics leaves out only a fallback program's compute stages.
- The push constant range is main's size again on a GPU without the fallback (NotRexed#640 added
  the fallback's two addresses to every layout, on every GPU).
- vertex_info gives stage 0's info for every program but a fallback one, as before.

Not tested on a GPU with mesh shaders: checked against main's code and by
tests/test_mesh_fallback.py (in the fallback's commit).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ys what is true now

The fallback is picked at every start, and the Insert menu (Graphics) can override it:

- Auto (the default): only on a GPU without mesh shaders (VK_EXT_mesh_shader with mesh and
  task shaders), which has buffer device addresses.
- Off: never, as RX_VK_NO_MESH_FALLBACK. On a GPU without mesh shaders the foliage, grass
  and many effects are then left out.
- On: on every GPU. A GPU's mesh shaders are then not turned on and the fallback draws
  instead. For AMD's RX 6000 and newer: Lichii NotRexed#814 found that native task and mesh draws
  hang those GPUs (a TDR on Windows, GPU resets with RADV on Linux), and the fallback
  avoids it. Not yet confirmed in Hookshot.

It applies after a restart (the next start prepares shaders again, the program cache's stamp
having the mesh mode), is saved as mesh_shader_fallback, and is not part of the Quality
preset: changing it keeps the preset and its VRAM caps.

The warning at launch is only for a GPU without mesh shaders that runs no fallback either.
It no longer says Hookshot needs an RTX 20 / RX 6000 / Arc card: it says whether the
fallback is turned off (the menu, or RX_VK_NO_MESH_FALLBACK) and how to turn it back on, or
that the driver lacks buffer device addresses (Vulkan 1.2), and to update it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t mesh shaders (RX 5500M)

With NotRexed#640 re-applied, an RX 5500M (RDNA1, 4 GB, no VK_EXT_mesh_shader) ran the fallback but
drew no foliage and froze at the first room with it. What was wrong, in the order found:

Translation (found by scripts/validate_spirv.py, or reading the emitter):
- A task-fed program read its payload's address from the push block by the byte offset (64)
  as a member index: every task-driven program wrote its payload to a garbage address.
- PhysicalStorageBuffer loads and stores had no Aligned (VUID 04708). Now LoadWord/StoreWord,
  Aligned in fallback stages only: native task and mesh stages are byte for byte main's.
- A task thread copied one word in 32 of its payload into its slot (the loop strode by the
  threads). The payload is now written once a task workgroup, its threads sharing the words,
  and its mesh workgroups read their task's.
- A mesh workgroup's index in its task (the payload's address 0, and its workgroup id) was
  its global slot, so it picked a meshlet past the end of its task's list. Now slot % task
  threads. The door of Link's house, the ivy and the vines drew from this point (with the
  draw-side fixes below in the same build).
- Strided's loop counter was a Function variable made at the end of main, and the task
  stored a bool into a u32 word: both invalid SPIR-V. The counter is Private now.

Draw side:
- A compute pipeline the driver refused was made again every draw, and leaked the stages
  made before it; and AMD took up to 12 s to compile one, in the draw: the freeze. They are
  compiled on a worker now (as compute programs are), the draw left out until they are made.
- Each program had record and payload buffers of its own, on each recording thread: 2.2 GB
  for 95 programs, VRAM full (the lock-up), and pairs that could not be made left their
  programs out for good (distant foliage white: its depth pass drawn, its colour pass not).
  A recording thread's programs share one pair now (64 MB of records, 8 MB of payload), a
  barrier before each draw's reset.

The floor grass:
- 1024 record slots a draw, each sized for the program's most primitives: the grass's 433
  tasks of 32 slots lost 13 of 14 meshlets. Now 16384 slots (a word each), and a mesh
  workgroup takes records for the corners it outputs from an allocator in the records'
  header, adding them to a VkDrawIndirectCommand there: the draw covers what was written
  (no empty vertices), primitives in the order taken (blended mesh geometry may draw out of
  order, as in Lichii's fallback).
- The grass is DrawMeshTasksIndirect: its task count is on the GPU, and the mesh stage was
  dispatched as groups * threads with groups 0, so no mesh workgroup ever ran. A started
  task thread now takes an index of the mesh dispatch (a VkDispatchIndirectCommand in the
  header) and writes its slot to the payload buffer's map; the mesh stage is dispatched
  from it and reads its slot from the map in the entry block (a load, no branch: later phis
  name the entry label).

At start:
- The door was white for a few seconds at spawn: its colour pass waited for its compute
  pipelines (AMD: 10-12 s each the first time). The precompile now makes every mesh fallback
  program's compute pipelines with the other compute programs, before play: the first start
  after a translator change or a driver update spends minutes there (its second count, about
  11000 on this setup, has the ~1350 fallback programs), later starts take them from the
  driver's cache in seconds. No white pop-in.

tests/test_mesh_fallback.py checks all of it (and that a GPU with mesh shaders draws as
main); its check that NotRexed#640's stage skip was there, which since matched another line by
chance, is gone.

Tested in play on an RX 5500M laptop (4 GB): foliage, floor grass and the door draw; all
1347 fallback programs pass spirv-val.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…as noise (working, further testing needed)

On a 4 GB RX 5500M, after an alt-tab (or a scene change) the textures turned to coloured
block noise: the path, the trees, Link's and the Kokiri's clothes, the minimap. Hookshot's
own bug, not the mesh fallback's; most likely on a card with little VRAM.

- With the game in the background the driver gave the process less VRAM, the texture budget
  shrank, and about a thousand textures were evicted in seconds. Made again when drawn,
  decoded ASTC is re-encoded to BC3 on the GPU through a scratch (an image and an output
  buffer, 160-288 MB), which is given back after 20 s unused.
- The scratch's output buffer could only be made in VRAM, within the budget
  (create_device_buffer): with the VRAM full it could not be made, and every BC3 upload
  failed ("no memory for the BC3 scratch" in the log).
- A failed upload was marked uploaded anyway, so as not to be tried every draw: a BC3
  image whose encode never ran was drawn with whatever its memory held, and was never made
  again unless the game rewrote its memory, which static textures never are.

Now:
- The scratch's output buffer goes to system memory when the VRAM is full, as the scratch
  image already did ("VRAM full: the BC3 scratch's output buffer ... in system memory").
- A BC3 texture whose upload failed stays not uploaded and is tried again 30 frames on
  (Texture::bc_retry_frame), the cause logged ("vk: BC3 texture #... its upload failed
  (...)": no staging memory, no memory for the BC3 scratch, ...).

Working in play on the RX 5500M after alt-tabbing, but tested briefly: needs more testing,
on other low-VRAM cards too. Between a failed upload and its retry a texture can still show
noise for a moment.

tests/test_bc_upload_retry.py checks the retry, its spacing, the logged cause and the
scratch buffer's fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/mesh_fallback_rdna1.md, for players and developers:

- Status, and the test laptop: an AMD RX 5500M with 4 GB of VRAM (3.1-3.4 GB usable),
  well under the 8 GB the readme recommends; the Minimum (4GB) preset for such cards, and
  the frame rates seen (about 15 fps docked, 19 with the preset, 34-38 after the fallback's
  later fixes).
- How it behaves: picked at every start; the Mesh shader fallback toggle (Auto by default,
  Off, On) and what On is for (AMD RX 6000 and newer that hang with native mesh shaders,
  Lichii NotRexed#814); the three counts of shader preparation, the third (about 11000 here) with
  the ~1350 fallback programs compiled before play so nothing pops in white, minutes on the
  first start after a change and seconds after.
- Why it was broken and how it was fixed, in the order found: NotRexed#640 reverted; invalid SPIR-V;
  the freeze; why the door and ivy drew before the floor grass; dark lighting and bloom on
  AMD; the lock-up and the white distant foliage on 4 GB; the grass (slots, then indirect
  draws that ran no mesh workgroup); the door white at spawn; NotRexed#640's broken native mesh
  pipelines behind the RTX 4060 Ti's device loss; the texture noise after alt-tab (working,
  further testing needed).
- Known limits, the tools and tests, and credits.

readme.md: cards without mesh shaders work through the fallback (tested on the RX 5500M,
4 GB); the requirements no longer ask for mesh shaders; the troubleshooting rows point to
the setting; the settings table and notes have Mesh shader fallback; links to the page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…debug switch

- The fallback's generated vertex stage starts every output generic at zero
  (point size 1), as the console's stage buffer does, instead of leaving
  components the mesh program does not write undefined.
- SHFL.BFLY in fragment stages takes Lichii NotRexed#273's broadcast fallback on
  AMD's proprietary driver, as the other shuffles do. The quad swap stays
  native: its programs have neither the lane-id input nor the Ballot
  capability the fallback needs (rerouting it crashed shader prep).
- RX_VK_WARP64=0|1 and RX_VK_FRAG_SHUFFLE_FALLBACK=0|1 override the warp
  profile for diagnosis (both are in the program cache's stamp).
- hookshot_fallback_debug.txt beside the exe hides fallback draws live:
  "off" hides all, "list" logs each program drawn, other lines are key
  prefixes to hide. Read at most every 500 ms; no file, no effect.

None of these fixes the glowing stems and flowers in Kokiri Forest; the
switch narrowed them to 19 fallback programs (see the doc).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds item 11 to docs/mesh_fallback_rdna1.md: the symptom, what was ruled
out, tried and kept, tried and reverted, and the 10 stem and 9 flower
programs the live switch traced the glow to. Neither is fixed yet; the next
step is a readback of a culprit program's records. Also documents
hookshot_fallback_debug.txt and the two RX_VK_* diagnostic switches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ve switch

Two new lines in hookshot_fallback_debug.txt:
- "read <prefix>": each program named copies its fallback records to a
  host buffer (16 slots of 4 MB), all in the frame after the file changes,
  so a depth prepass and its colour pass are compared with the wind the
  same. Once the GPU's timeline passes the recording, each record word's
  range, NaNs, infinities and denormals are logged, and the records are
  written to hookshot_fallback_read_<key>.bin.
- "info <prefix>": the next draw of each program named logs its targets
  and formats, each target's game mask, shader components and final write
  mask, blending, and the depth and stencil test.

timeline_completed() (vk_replay.cpp) exposes the timeline value the GPU
has reached. No file, or no such lines: nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd (bright red/green foliage stems)

(cherry picked from commit deaf4ff)
…e new switch lines

Item 11 now has the cause. The "stem" culprits were depth passes; the stems'
colour pass computed its light-probe lighting wrong because of a translator
bug on every GPU (upstream NotRexed#757, NotRexed#874, NotRexed#986), which upstream PR NotRexed#759 fixed.
Positions and triangles read back in one frame were bit-identical. Also
notes the hearts glitch as upstream NotRexed#993 and documents "read" and "info".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lost GPU-drawn textures

The 97% rule set the budget below the textures held every 10 frames while
the driver's count was at its limit. Evicting frees space inside VMA's
blocks, which the driver still counts, so the count stayed there and the
budget shrank again: a ratchet. On a 4 GB RX 5500M (3337 of 3337 MB used,
1224 MB free in the blocks) a paused game, whose scene draws nothing, lost
1700 MB of textures in seconds and made them again on unpausing (a long
wait). The HUD's GPU-drawn pieces went too and came back as black glyphs
until an area change.

Now that rule waits while the blocks hold an eighth of the allowance free;
the rule that judges pressure net of that free space still evicts. The
VRAM log line says "not squeezed" then. RX_VK_SQUEEZE_ALWAYS=1: as before.

Also logs each small GPU-drawn texture evicted with its texels lost, and
why the keep for finalized ones passed it over (once a size and format).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Black hearts, the minimap glitch and the slow unpause on 4 GB cards: the
cause and the fix, needing more testing. Known limits now say NotRexed#759 is
cherry-picked onto a branch 170 commits behind upstream main (with
upstream's open AMD stripes report), and that stutters are not diagnosed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the BC3 path re-encodes an ASTC texture array, it decoded and encoded
up to kBcLayers consecutive layers in one run. In the smaller levels, every
layer after the run's first came out wrong. Single-layer textures were
unaffected. The terrain's material arrays (1365 x 1365 x 90, ASTC 8x8) are
sampled at those levels from a distance, so Hyrule Field's ground showed
its tiles outlined in dark lines that went away up close.

On an RX 6600 the lines went away with one layer a run, with the arrays
kept as RGBA8, and with samplers capped at mip 0. One layer a run keeps
the arrays BC3, so there is no VRAM cost (RGBA8 cost ~1.3 GB, too much for
an 8 GB card). RX_VK_BC_LAYERS=<n>: up to n layers a run, as before with 8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit ec89f92)
…s 7x GPU time and rainbow ground on AMD

Issue 823's RX 6700 XT logs: v2.0.4 played at 10-13 ms of GPU time a frame, v2.0.5 at 70-100 ms, the only AMD
render change between them being NotRexed#705 (the 1365 x 1365 x 90 terrain material arrays re-encoded at 1536 x 1536
with 8 of their 11 levels). AMD players also posted rainbow/iridescent ground and characters on v2.0.5.
RX_VK_UNEVEN_BC=0 turns it on (any card) for testing; by default BC3 is made as in v2.0.4 (the thin grid on
Hyrule Field's terrain seams comes back on AMD until the resize is fixed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit dc2f4dc)
…ning hitches

The two upstream v2.0.6 texture fixes cherry-picked before this (the AMD
uneven-BC3 resize off by default, BC3 array re-encode one layer a run):
symptom, cause and the in-game result on the RX 5500M. Known limits now
name the 2 s re-upload of the 12288x12288 shadow map instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rakefire-impala rakefire-impala changed the title Working fallback mesh shaders AMD RDNA1 Working fallback mesh shaders AMD RDNA1 and various performance and accuracy fixes Nov 9, 2026
@rakefire-impala rakefire-impala changed the title Working fallback mesh shaders AMD RDNA1 and various performance and accuracy fixes Working fallback mesh shaders (AMD RDNA1) and various performance and accuracy fixes Nov 9, 2026
@NotRexed

Copy link
Copy Markdown
Owner

Thanks for all the work on the RDNA1 fallback. The fallback itself is on main, the v2.0.12 branch carries its crash fixes (#1314, #1317), and #1383 is next. This branch conflicts with main in 10 files and mixes many changes. Closing; any fix not covered by those would be very welcome as a small PR of its own.

@rakefire-impala

Copy link
Copy Markdown
Contributor Author

Thanks, that makes sense. I've split what main doesn't already cover into small PRs, each fixing one thing, all on current main:

They merge cleanly with main and with each other. #1505–#1507 were tested on an RX 5500M (RDNA1, 4 GB, no mesh shaders). #1507 changes the graphics translation on AMD cards that can't pin graphics subgroups, so an RDNA2+ tester would help before merging it.

@CiLiBi18

Copy link
Copy Markdown
Contributor

Tested on a GTX 1070 (Pascal, no mesh shaders) and it brings the missing geometry back.

Setup: this PR's head (16229ef) built as-is on Linux, GTX 1070 8 GB / NVIDIA 580.178.04, i7-7700HQ, output 1920x1080 at 75%. The comparison is v2.0.11 + #1314 + #1317, played through the same part of the game the night before.

  • Geometry: on v2.0.11 + vk: land #792's four mesh fallback fixes (the fallback's compute pipelines crashed NVIDIA's Pascal driver) #1314/vk: a mesh fallback program whose compute pipeline failed is not compiled again #1317, some mesh-shader geometry ran through the compute fallback but wasn't drawn, while its collision stayed. Inside the Deku Tree, the walls and the block you push to climb up and go on were invisible, which made the dungeon very hard to follow. With this PR they are all drawn. I haven't checked Link's house door yet.
  • Stability: 0 driver faulted, 0 failed fallback pipelines. The only failed programs are the 10 movie-upscaler networks, which are left out on purpose on this card.
  • Frame rate, gameplay only: mostly 30–40 fps, with GPU time 15–18 ms and submit 12–20 ms a frame, at 500–700 draws and ~90 fallback dispatches. It isn't a like-for-like comparison with my v2.0.11 runs (median 25 fps), because the scenes weren't the same.
  • Stalls: the first two minutes in a new area had seven submits of 1.7–9.8 s, almost all of it in draw. One fallback compute pipeline took 101 s to compile on a worker: fallback: the compute pipeline of 55d1da3fea1603d4 stage 1: SLOW compile, 101648 ms (on a worker). The 9.8 s stall came right after it. This was a fresh game folder with an empty pipeline cache (precompile: ... 0 graphics), so part of this is first-time compiling. After those two minutes, I saw no more stalls.

One more data point: the storage buffer NOT bound (reads zeros) lines for the far-foliage programs (68642bb810ee3e7c, 51183fe750d5d0c8, d905f52b7517c681…, at c5+0x60 / c5+0x1d0) still show up here and reach the log's cap of 40. So they weren't what hid the geometry on this card.

Happy to run anything else on Pascal if it helps get this merged.

@rakefire-impala

Copy link
Copy Markdown
Contributor Author

Thanks for testing this on Pascal. It's the first report from a second GPU vendor, and the Deku Tree result is exactly the kind of thing this needed.

#1023 was closed so its fixes could go in as small PRs. The parts you tested are now:

All of them merge cleanly with main and with each other. If you could run main + #1509-#1513 (and #1581 if you like) on the GTX 1070 and post what you see on #1510 or #1511, that would be the strongest case for merging them.

@CiLiBi18

Copy link
Copy Markdown
Contributor

Thanks for splitting it up. I'll build main (88b554e) + #1509–#1513 + #1581 as they are on the GTX 1070 (they merge cleanly here too) and report on #1510. I'll check:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants