Skip to content

vk: RX_VK_MESH_NOCULL, the Steam Deck's task shader hang without leaving models out (#817) - #1300

Merged
hckrman101 merged 1 commit into
NotRexed:mainfrom
KewinBR:deck-mesh-nocull
Nov 9, 2026
Merged

hckrman101 merged 1 commit into
NotRexed:mainfrom
KewinBR:deck-mesh-nocull

Conversation

@KewinBR

@KewinBR KewinBR commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

What

A new renderer switch, RX_VK_MESH_NOCULL=<task shader code key prefix,...>, turns off the culling inside the task shaders it names. On a Steam Deck, RX_VK_MESH_NOCULL=2c56dc1006942f74 avoids the GPU hang of #817 with every model drawn, where RX_VK_SKIP_CODE=2c56dc1006942f74 (#1165) avoids it by leaving out the ~30 programs that use that task shader, and the houses, platforms and rupees they draw.

  • gpu/vk/vk_shader.cpp: mesh_nocull() runs on the translated task program, before dump_ir. It looks for INotEqual(BitwiseOr32(x, BitwiseAnd32(4, GetCbufU32(3, 2320))), 0), replaces its uses with true and runs DeadCodeEliminationPass. If the pattern is not found, the program is left as it is and the log says so. It logs the first 8 task shaders it met.
  • readme.md: the Deck troubleshooting row now points to this switch first, and the switch is described under Switches for testing, next to RX_VK_SKIP_CODE.

Why it works

Task shader 2c56dc1006942f74 projects each meshlet's bounding sphere onto a grid of c3[3088] x c3[3092] cells and, for every covered cell, reads four words from c3[3120 + cell * 16] by a dynamic index (two nested loops). On RDNA 2 some draws run for seconds, and past 10 s the kernel resets the ring. The shader's answer is:

visible = (result & 1) | (c3[2320] & 4)

Bit 4 of c3[2320] is the game's own "draw everything" switch. Making that comparison true lets dead code elimination take the grid walk with it: no loops are left, every meshlet goes to the mesh stage, and the rasterizer clips what is off screen. It costs some GPU time on cards that cull quickly, so it is off unless set. The name is under RX_VK_MESH, so the program cache starts again when it changes.

Tested

Steam Deck (RADV VANGOGH), SteamOS, Game Mode, HOS_HANDHELD=1, RX_VK_MESH_OPT=1, the memory caps from #817, with the same change on by default in a local build:

  • v2.0.7: 15 minutes driven through Kokiri Forest (scripted input), 0 stalls, 0 draws left out, about 30 fps. Before it, the same route hung within seconds. A player then went from the start to the Kokiri Sword in Game Mode: the houses, the platforms over the pond and the rupees that RX_VK_SKIP_CODE hides are all there.
  • v2.0.11: 12 minutes turning the camera and walking in Kokiri Forest, 0 stalls, 0 device lost.
  • This branch (main with the switch off by default) builds with the Linux AppImage toolchain, and tests/run_all.py gives the same results as main without the change. I did not run this exact build on the Deck: the runs above are the same function, on by default.

The flashing on integrated GPUs (#1186, #1126) is a separate problem and is not touched here.

Generated with Claude Code

…ing models out (issue 817)

OoT's meshlet culling task shader 2c56dc1006942f74 walks a coarse depth grid cell by cell for every meshlet; on
RDNA 2 (Steam Deck) one draw runs past the kernel's 10 s and the GPU is reset. Its answer is
visible = (result & 1) | (c3[2320] & 4), bit 4 being the game's own "draw everything". With the switch set, that
comparison becomes true and dead code elimination removes the grid walk: every meshlet is drawn, none is left out
(RX_VK_SKIP_CODE leaves the programs out, and their models with them). Off unless set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant