[rakefire-impala]

Fixes dark lighting and white-glowing objects (a basket in Link's house, bloom) on AMD's Windows driver. Two causes: 64-lane graphics waves that the translator treated as 32-lane, and AMD's broken fragment-stage subgroup shuffles (Lichii #273).

Part of splitting up #1023. Needs testers with RDNA2 or newer: I only have RDNA1.

Problem

• Warp size. AMD lets only compute stages pin their subgroups to NVN's 32 lanes (requiredSubgroupSizeStages = 0x20 on the RX 5500M). Fragment and vertex stages run at 64. The translator assumed 32 lanes everywhere (warp_size_potentially_larger_than_guest false), so the ballots and shuffles in the game's fragment shaders ran across both halves of a 64-lane wave, giving dark lighting and glowing bloom.
• Shuffles. AMD's proprietary driver gets fragment-stage subgroup shuffles wrong. Lichii AMD RX6600 , I5-11400f , 16gb ram Crash after intro #273 found it (the white basket) and routes them through broadcasts.

Change

• make_profile sets warp_size_potentially_larger_than_guest, and max_subgroup_size to the device's largest, only when a graphics stage can't be pinned and the default is wider than 32. Eden's long path then keeps each 32-lane guest warp to its half. NVIDIA (32 lanes) is unchanged.
• Per stage (profile_graphics): a graphics stage the GPU does pin to 32 lanes (pin_subgroup) keeps main's short path, so only stages that really run at 64 lanes change:
  • RDNA1 (0x20, compute only): vertex and fragment take the long path.
  • RDNA2/RDNA3 (0xf0, reported from an RX 7800 XT in this thread): only the vertex stage. Fragment, task and mesh keep main's translation.
• Compute keeps the short path (profile_pinned), including the mesh fallback's compute stages, because they are pinned to 32. Giving every compute program the long path changed its SPIR-V, missed the driver's cache, and AMD recompiled shaders for hours at start.
• On AMD's proprietary driver, fragment-stage shuffles take the broadcast fallback (has_broken_spirv_fragment_shuffle), and so does SHFL.BFLY. The quad swap stays native: its programs have neither the lane-id input nor the Ballot capability the fallback needs, and rerouting it crashed shader preparation.
• The program cache's stamp includes the profile's warp choices and the pinned stages, so a cache translated for another GPU's warps isn't used.
• Diagnostics: RX_VK_WARP64=0|1 and RX_VK_FRAG_SHUFFLE_FALLBACK=0|1 override both, whatever the GPU, to help bisect reports.

The depth-compare part of #273 is already on main (#834); only the test checks it here.

Risk

On RDNA1, this changes the vertex and fragment programs' translation, so those players re-translate on their first start. On RDNA2/RDNA3, only vertex programs change (plus the shuffle fallback below), because their other graphics stages are pinned to 32 lanes. An RX 6000/7000 result would still be welcome before merging. NVIDIA and Intel are unaffected by the warp part. The shuffle part is limited to VK_DRIVER_ID_AMD_PROPRIETARY.

Tests

tests/test_amd_fragment_warp.py (new) checks the profile, the per-stage choice (pinned stages keep the short path; RX_VK_WARP64 still sets them all), the pinned profile's uses, the stamp, the shuffle fallback and the depth-compare rule.

Tested on

GPU	AMD Radeon RX 5500M (RDNA1, 4 GB VRAM), laptop
Driver	AMD Windows driver, Windows 11
Result	Lighting and bloom correct; the white basket fixed (tested on an earlier base; rebased on current main, game-free tests pass)
RDNA2+	Not tested

[lucasmazzoni45-del]

Data point for the testers asked for here, from an RX 7800 XT (RDNA3), Windows 11, driver 0x8001a0, v2.0.12:

The log line is vk: subgroups: default 64 lanes, 32-64, pinned to 32 on stages 0xf0. By the Vulkan stage bits that is fragment (0x10), compute (0x20), task (0x40) and mesh (0x80): on this card the fragment stage can be pinned to 32 lanes, and only the vertex stage cannot. So the "64-lane graphics warps" part of this PR may not apply to RDNA3 at all; the broken fragment-stage shuffles of AMD's proprietary driver (Lichii #273) might still be relevant. On the RX 5500M the value is 0x20 (compute only), which is why it matters there.

I have not tested this PR on this card. If it helps I can build it on top of v2.0.12 and report the first-start preparation time (the PR says graphics programs are translated again on AMD), whether the picture changes anywhere, and the log lines, using RX_VK_WARP64=0|1 and RX_VK_FRAG_SHUFFLE_FALLBACK=0|1 to separate the two parts.

[rakefire-impala]

Thanks, that's exactly the data this needs.

On your card this PR does turn the 64-lane path on. The condition is subgroup_size > 32 and not every graphics stage pinnable to 32 lanes ((sg32_stages & VK_SHADER_STAGE_ALL_GRAPHICS) != VK_SHADER_STAGE_ALL_GRAPHICS), and with 0xf0 the vertex stage isn't. So on RDNA3 the vertex and fragment stages both get the long warp path, the fragment stage even though it runs at 32 lanes there. The long path is still correct at 32 lanes, so this should cost translation time, not correctness. Narrowing it to the stages that can't be pinned is a reasonable follow-up, and your results would show whether it's worth it.

The two parts are independent. The most useful runs, set in hookshot_env.txt:

Run	RX_VK_WARP64	RX_VK_FRAG_SHUFFLE_FALLBACK	Shows
1	0	0	main's behaviour, the baseline
2	0	1	Lichii #273's shuffle fallback alone
3	(unset)	(unset)	both parts, this PR's default

For each run: the first-start preparation time, and whether lighting, bloom or anything else looks different. On the RX 5500M it showed most in Kokiri Forest and in Link's house (a basket drew white). Both switches are in the program cache's stamp, so each run re-translates once.

[lucasmazzoni45-del]

First result for run 3 (both parts, the PR's default) on an RX 7800 XT (RDNA3), Windows 11, driver 0x8001a0, RX_VK_WARP64 and RX_VK_FRAG_SHUFFLE_FALLBACK unset, RX_VK_PRESENT_QUEUE=1. Thanks for the correction above: you are right that the long warp path is on here (the vertex stage is not in 0xf0); my earlier note that it might not apply was wrong.

Build: v2.0.12 plus this PR (merged with no conflicts), plus PR #1130 at 6f76660 and two local memory edits for AMD (image pool preallocation, larger capture buffer classes), so it is not a clean main + PR build.

First start: prepared 38809 shader programs (46556 new, 111 failed, ...) in 65.4 s (111 failed is the same count as without the PR); the log's total translation time was 475 s across threads; from launch to a steady 60 fps at the title (headless, including the driver's pipelines) it took about 5 minutes.

What I saw (normal play, about 12 minutes): lighting and bloom looked normal in Kokiri Forest, in Link's house and in front of Ganon's Castle as adult Link. No white-glowing objects, no too-dark areas, no errors, no device lost. I did not see those problems on this card before the PR either, so "normal" shows no regression, not that the PR fixes anything here.

Frame times (not conclusive): mean 57.7 game fps (median 60, 5% worst 46), GPU 4.9 ms a frame on average, p95 9.5 ms. 28 submits of 40 ms or more (15 of 100 ms or more, longest 1.4 s) against 12 in a 9.6-minute session on the same GPU without this PR; the dominant part of the long ones is draw in both, and the routes differ (this one had more area loads), so I am not reading it as a cost of the PR yet.

Not done yet: runs 1 (0/0, main's behaviour) and 2 (0/1), on the same route, to separate the two parts and the frame times. I will post them here.

[rakefire-impala]

Following your point, I've pushed a narrower version (6943ac0). The 64-lane path now applies per stage, only to the graphics stages the GPU can't pin to 32 lanes. On your RX 7800 XT that means only the vertex stage changes; fragment, task and mesh keep main's translation, while the AMD fragment shuffle fallback still applies.

The test plan stays the same. In run 3 (both parts on), only your vertex programs are re-translated with the long path, so the first-start time should come out close to run 2's.

[lucasmazzoni45-del]

Run 1 (RX_VK_WARP64=0, RX_VK_FRAG_SHUFFLE_FALLBACK=0, main's behaviour) on the RX 7800 XT, same build and settings as my run 3 above (RX_VK_PRESENT_QUEUE=1), same route (Kokiri Forest, Link's house, in front of Ganon's Castle as adult). This is the head before your narrower 6943ac0: my run 3 was on dae962a, so run 3 has to be redone; I will rebuild on 6943ac0 and post runs 2 and 3 with it.

Picture: normal in all three places in both runs. Nothing white, nothing too dark, no errors, no device lost.

First start: prepared 38809 shader programs ... in 58.4 s with 10091 stage libraries; run 3 on dae962a was 65.4 s with 0 stage libraries listed in that line.

Frame times over the same first 26,400 presents (about 7.3 minutes), one run each, run 3 done first:

	run 3 (dae962a)	run 1 (0/0)
game fps mean / 5% worst	56.9 / 45.1	58.7 / 51.8
5-second samples under 50 fps	19 of 144	6 of 165
GPU ms a frame, mean / p95	6.0 / 10.8	5.9 / 9.6
submits of 40 ms or more	28	23
of 100 ms or more (longest)	15 (1.4 s)	7 (3.2 s)
SLOW driver compile lines in play	21	5

I would not read this as a cost of the PR: one run each, the order was fixed, and the PR changes the translated shaders, so run 3 also compiled many pipelines for the first time in play (21 against 5 slow driver compiles), which can explain part of the hitches. I cannot separate that from a real cost with these two runs. The longest hitch is the other way round (3.2 s in run 1).

Thanks for narrowing it per stage. For a fair comparison I will repeat run 3 on 6943ac0 after a warm-up pass, and add run 2 (0 / 1).

[rakefire-impala]

Thanks for runs 1 and 3. #1507 has merged since, including the per-stage change (6943ac0, on main as c433687), so it's main's default now.

Your run 3 had more long submits than run 1 (15 against 7 of 100 ms or more) and more slow driver compiles (21 against 5). But it was on the wider version, which put the long warp path on every fragment shader with warp operations. On your card the merged version gives it only to the vertex stage, and only 59 of the 17,233 distinct vertex shaders in my program cache use warp operations.

The other part that could still cost RDNA3 something is the fragment shuffle fallback. On AMD's Windows driver it turns each fragment-shader shuffle into 32 broadcasts, and pinning the fragment stage to 32 lanes doesn't turn it off. Your run 1 had it off and the picture was normal, and the white basket it fixes has only been seen on an RDNA1 card so far. So it may only be needed where the fragment stage runs at 64 lanes.

Could you test that on current main? No PR build is needed. Use the same route both times, each after a warm-up pass. The switch is in the program cache's stamp, so changing it re-translates once.

Run	hookshot_env.txt	What it is
A	neither switch set	main as it ships
B	RX_VK_FRAG_SHUFFLE_FALLBACK=0	the shuffle fallback off, the vertex path still on

For each run, the most useful numbers are the frame times (game fps mean and 5% worst, submits of 100 ms or more) and the count of SLOW driver compile lines. Please also note anything that looks different, especially Link's house (the basket) and Kokiri Forest's plants. If B is faster and draws the same, I'll limit the fallback to fragment stages that run at 64 lanes. That keeps it for RDNA1 and leaves RDNA2/3 as they were before #1507.