fix(vk): a fence wait covers every part of a split submission, not only the first by willcrisis · Pull Request #2419 · NotRexed/Hookshot

willcrisis · 2026-11-11T18:10:58Z

On AMD integrated GPUs (ROG Ally, Legion Go, Steam Deck) the picture breaks for one frame at a time: the HUD drawn stretched across the screen, grass flat white, the whole screen flashing. The game rewrites memory the GPU is still reading, because its fence waits return after the first part of a submission.

Cause

A submission longer than RX_VK_CHUNK draws (default 120), or one stopped at an event wait and finished later, is recorded in several recordings that all carry its value. submit_recording maps the value to its recording with value_rec.emplace, which keeps the first one: when the last part finishes the submission, the key is already there and nothing changes. So wait_recorded (the game's fence waits in wait(), and the exposure readback wait) returns once the first ~120 draws are done.

With no VRAM mirror of the buffer pools (integrated GPUs), the later parts read guest memory in place, and the game has by then written the next frame's uniforms, vertices and exposure meter into it. The HUD is drawn late in the frame, so it gets the wrong transforms most often; the stale exposure readback gives the whole-screen flashes.

myylkn found the same root cause on a Radeon 8060S (#997, with a patch in a comment). This PR is a variant of that patch:

find and an update in place rather than value_rec[v], so it builds against tests/draw_stream_visibility_test.cpp, which models value_rec as a std::multimap.
It keeps registering r.value when the recording does not signal (the parts split off before the last), raised to the latest seq. A wait on a stopped submission still finds its recording, instead of falling back to the timeline wait, which forces stopped work past its event waits.

Change

gpu/vk/vk_replay.cpp, submit_recording: each value maps to the latest recording that carries it.
tests/test_value_rec_last_part.py: runs the statements cut from submit_recording, for a submission split in three parts and for a stopped submission finished in a later recording. It fails on main (value 10 -> recording 1 (expected 3), value 11 -> recording 4 (expected 5)) and passes with the change.
docs/tested-configurations.md: the three runs below.

Tested

ROG Ally X (Ryzen Z1 Extreme, RADV PHOENIX, Mesa 26.2.3), Linux (Anatase, Fedora-based), counting one-frame luminance spikes in screen recordings (a frame brighter or darker than both its neighbours):

v2.0.13, Hyrule Field	One-frame spikes	15/255 or more	`GPU waits in 5 s`
without the change (69 s)	104	15	4–109 sync waits, 31–232 ms
with the change (42 s)	3	0 (the largest is a sword hit)	103–137 sync waits, 1.4–1.8 s

Then about a day and a half of play with the change, rebuilt for v2.0.14 and v2.1.0 (the renderer alone, RX_VK_ONLY=1): no flashes or stray HUD seen in Hyrule Field, Death Mountain Trail and Goron City, in Desktop Mode and in Steam's Gaming Mode. A 57 s Gaming Mode recording on v2.0.14 had no one-frame spikes at all.

It costs frame rate on this device. The FPS overlay in the field read roughly 45–70 before and 38–49 after (two recordings, not a controlled benchmark): part of the old frame rate was the game running ahead of the GPU. The waits now last as long as the GPU work they stand for.

python -I -B tests/run_all.py with the Vulkan headers: the same results on a clean checkout of main with and without the change, apart from the new test, which passes. test_draw_stream_visibility, test_capture_slot_ownership and test_parallel_rb_refresh, which compile submit_recording, pass.

Not tested: Windows, NVIDIA and discrete AMD cards (the early wait is there too, but their VRAM mirror is synced at the submit, which should hide it; not measured), RX_VK_PARALLEL segments (off by default; the same mapping applies to them), and the Steam Deck.

Part of [Tracking] AMD handhelds and APUs (ROG Ally / Ally X, Legion Go, Radeon 780M/860M/890M): black screen, device lost, reboot #997 (AMD handhelds and APUs tracker).
fix(vk): HUD and lighting artifacts on GPUs without buffer mirrors #1795 copies uniform buffers on GPUs without mirrors for the same symptoms. The two can coexist: that one keeps a late write from reaching the GPU; this one makes the fence waits cover the work they stand for. An A/B of fix(vk): HUD and lighting artifacts on GPUs without buffer mirrors #1795 with this change in and RX_VK_UB_SNAPSHOT=0 would show how much of its effect was this wait.
fix(vk): with per-queue fences, a CPU read-back waits for the GPU work that writes it again #2125 (merged) lets stopped work finish before a CPU readback. Complementary: the readback's wait that follows goes through value_rec, which this corrects.

Screenshots

Before (v2.0.13, Hyrule Field and Castle Town; each strip is the frame before, the broken frame, the frame after):

After (v2.0.13 with the change, Hyrule Field, fights included; the last strip is the largest one-frame spike left, a sword hit):

The analysis and the change were made with AI assistance (Claude); the testing and recordings are mine.

🤖 Generated with Claude Code

…ly the first On AMD integrated GPUs (ROG Ally, Legion Go, Steam Deck) the HUD is drawn stretched across the screen for a frame, grass glows white and the whole screen flashes, because the game rewrites memory the GPU is still reading: its fence waits return after the first part of a submission. A submission longer than RX_VK_CHUNK draws (default 120), or one stopped at an event wait, is recorded in several recordings that all carry its value. submit_recording kept the first one's seq (value_rec.emplace), so wait_recorded -- the game's fence waits and the exposure readback wait -- ended once the first part was done. Without a VRAM mirror the later parts read guest memory in place, which the game had already filled with the next frame's uniforms, vertices and exposure meter. Each value now maps to the latest recording that carries it. tests/test_value_rec_last_part.py runs the statements cut from submit_recording; it fails on the old code (the split and the stopped case). See NotRexed#997 Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts: # docs/tested-configurations.md

Joey-Magnus

This looks right to me, and it's a well-made PR: a clear cause, a fixture cut from the real code, and before/after recordings on hardware. I confirmed that emplace keeps the first part's seq, and that maybe_split (gpu/vk/vk_replay.cpp:1321), the parallel segments (:1480) and finish_recording (:1892) all submit recordings that carry the same r.value. Before this change, wait_recorded therefore waited on the first part. I ran test_value_rec_last_part, test_draw_stream_visibility, test_capture_slot_ownership and test_parallel_rb_refresh on the PR's head, and they pass. As a negative control I pointed the new test at main's vk_replay.cpp with --source: it fails the two cases you list. The frame-rate cost is honest and expected, since the waits now cover the work they stand for.

question gpu/vk/vk_replay.cpp:382-388 and :2143-2150: a window remains while a submission is still being recorded. Once the first part has gone through maybe_split, value_rec[v] is that part's seq until the last part is submitted. wait() doesn't take replay_mu, and q_last/value_queue are set before submit_job (:4261). So another game thread that waits on that queue's fence in that window still has mine true (the queue isn't blocked) and returns after the first part only. The same holds after a stopped submission is let go and while its remainder is being recorded. This is much narrower than before, when the mapping stayed wrong for good, so I don't think it should hold the PR up. If you want to close it, one option is to mark the value as "still recording" (for example, a set under rec_map_mu that run_submit fills while it records and finish_recording clears). wait_recorded would then return false for a marked value and fall back to the timeline wait. If it isn't worth it, a one-line note in the comment that the map is final only once the last part is submitted would help the next reader.

minor: the PR conflicts with main in docs/tested-configurations.md only (rows added at the top since). The code merges cleanly.

A dGPU has its own fast memory and a slower bus to system RAM; stock mirrors every guest pool into VRAM and keeps both copies coherent with captures, shadows and write-backs. At 4K that coherence is CPU-bound: the write-back diff compares ~3.1 GB per 600 submits and mirror syncs move ~137k dirty pages, for 35.6 fps where 1080p runs 60 (issue NotRexed#2099). Under RX_VK_DGPU=1 a pool's first binding makes no mirror (MIRROR_REFUSED, the state iGPU runs use today): reads take the lock-free TLS path and GPU reads/writes land in the imported guest memory, coherent by construction - no mirror sync, no write-back. Every 600-submit window the heaviest guest-memory-read pool with >=1 MB of reads and not write-dominated runs a measured trial (120 frames direct -> 12 settle, held until the mirror is really on -> 120 mirrored, on the recordings' own timestamps); the mirror stays only when the win is both >10% (RX_VK_DGPU_GAIN) and >0.25 ms (RX_VK_DGPU_FLOOR) on present-to-present frame time, so the verdict sees the mirror's CPU cost and its PCIe benefit alike. Trials whose windows did not measure the same workload (zero binds, a >2x bind shift, a mirror that never came) abort with no verdict and no skip line; a pool that aborts three times sits out the session, and iGPUs run no trials. A lost verdict persists to hookshot_mirror_skip.txt beside the game's own state files, keyed to the device and driver (a foreign or header-less file is ignored and re-trialled). RX_VK_DGPU_TRIALS=0 is pure direct. Trials need serial replay (documented); under RX_VK_PARALLEL the policy runs trial-less. Unset = stock exactly: the added counters and logging are gated on the switch. Refused pools read uniform buffers as submitted (NotRexed#1795's per-pool form: the snapshot engages for mirror-less pools at the read, trial-won pools keep in-place mirror binds), so the artifact class the issue documents on mirror-less GPUs is covered under the policy too. Measured on RX 9070 XT / RADV, 4K, Uncapped frame cap, warp-normalized adjacent pairs (Hyrule Field): 70.2->78.0, 71.0->77.0, 59.6->76.6 fps (+9.7% median), draw parity 705 vs 709, GPU 9.2 ms of a ~14 ms frame. Mirror-sync dirty pages 136,872 -> 0; write-back diff 3.06-3.13 GB -> 330 MB or less per 600 submits. Review round 2 (external): RX_VK_DGPU_TRIALS=0's predicate was inverted and now matches the docs (=0 is pure direct); the window's read counter moved off a shared map into the pool (dgpu_read, both direct paths); Settle holds until the mirror is actually live; zero-bind baselines and >2x bind drift between the direct and mirrored windows abort the trial un-blacklisted; keep_line refuses signed and trailing-garbage lines. Review round 3 (hckrman101, Joey-Magnus): refused pools read UBs as submitted (NotRexed#1795 per-pool, live-smoked at 1.26M copies fault-free); verdicts decide on present-to-present frame time instead of GPU-only time (the mirror's CPU and upload-buffer cost is what the frame pays); the policy runs trial-less under RX_VK_PARALLEL and its fast-path counters are parallel-safe; no trials on iGPUs and a 3-abort cap; the skip list carries a device+driver header and ignores foreign files; the test driver uses {cxx}. Rebased onto main's env_on switch rule (NotRexed#2042) and the split-submission fence work (NotRexed#2419). Co-authored-by: GLM-5.3-Flash <glm-5.3-flash@opencode.noreply.github.com>

A dGPU has its own fast memory and a slower bus to system RAM; stock mirrors every guest pool into VRAM and keeps both copies coherent with captures, shadows and write-backs. At 4K that coherence is CPU-bound: the write-back diff compares ~3.1 GB per 600 submits and mirror syncs move ~137k dirty pages, for 35.6 fps where 1080p runs 60 (issue NotRexed#2099). Under RX_VK_DGPU=1 a pool's first binding makes no mirror (MIRROR_REFUSED, the state iGPU runs use today): reads take the lock-free TLS path and GPU reads/writes land in the imported guest memory, coherent by construction - no mirror sync, no write-back. Every 600-submit window the heaviest guest-memory-read pool with >=1 MB of reads and not write-dominated runs a measured trial (120 frames direct -> 12 settle, held until the mirror is really on -> 120 mirrored, on the recordings' own timestamps); the mirror stays only when the win is both >10% (RX_VK_DGPU_GAIN) and >0.25 ms (RX_VK_DGPU_FLOOR) on present-to-present frame time, so the verdict sees the mirror's CPU cost and its PCIe benefit alike. Trials whose windows did not measure the same workload (zero binds, a >2x bind shift, a mirror that never came) abort with no verdict and no skip line; a pool that aborts three times sits out the session, and iGPUs run no trials. A lost verdict persists to hookshot_mirror_skip.txt beside the game's own state files, keyed to the device and driver (a foreign or header-less file is ignored and re-trialled). RX_VK_DGPU_TRIALS=0 is pure direct. Trials need serial replay (documented); under RX_VK_PARALLEL the policy runs trial-less. Unset = stock exactly: the added counters and logging are gated on the switch. Refused pools read uniform buffers as submitted (NotRexed#1795's per-pool form: the snapshot engages for mirror-less pools at the read, trial-won pools keep in-place mirror binds), so the artifact class the issue documents on mirror-less GPUs is covered under the policy too. Measured on RX 9070 XT / RADV, 4K, Uncapped frame cap, warp-normalized adjacent pairs (Hyrule Field): 70.2->78.0, 71.0->77.0, 59.6->76.6 fps (+9.7% median), draw parity 705 vs 709, GPU 9.2 ms of a ~14 ms frame. Mirror-sync dirty pages 136,872 -> 0; write-back diff 3.06-3.13 GB -> 330 MB or less per 600 submits. Review round 2 (external): RX_VK_DGPU_TRIALS=0's predicate was inverted and now matches the docs (=0 is pure direct); the window's read counter moved off a shared map into the pool (dgpu_read, both direct paths); Settle holds until the mirror is actually live; zero-bind baselines and >2x bind drift between the direct and mirrored windows abort the trial un-blacklisted; keep_line refuses signed and trailing-garbage lines. Review round 3 (hckrman101, Joey-Magnus): refused pools read UBs as submitted (NotRexed#1795 per-pool, live-smoked at 1.26M copies fault-free); verdicts decide on present-to-present frame time instead of GPU-only time (the mirror's CPU and upload-buffer cost is what the frame pays); the policy runs trial-less under RX_VK_PARALLEL and its fast-path counters are parallel-safe; no trials on iGPUs and a 3-abort cap; the skip list carries a device+driver header and ignores foreign files; the test driver uses {cxx}. Rebased onto main's env_on switch rule (NotRexed#2042) and the split-submission fence work (NotRexed#2419); the trials' functions sit outside the write-back observation's compile-span check (NotRexed#2131) and the GCN mirror-less default (NotRexed#2343) composes unchanged (on GCN the switch runs trial-less direct plus the UB snapshot). Co-authored-by: GLM-5.3-Flash <glm-5.3-flash@opencode.noreply.github.com>

…ly the first (NotRexed#2419) On AMD integrated GPUs (ROG Ally, Legion Go, Steam Deck) the picture breaks for one frame at a time: the HUD drawn stretched across the screen, grass flat white, the whole screen flashing. The game rewrites memory the GPU is still reading, because its fence waits return after the first part of a submission. ## Cause A submission longer than `RX_VK_CHUNK` draws (default 120), or one stopped at an event wait and finished later, is recorded in several recordings that all carry its value. `submit_recording` maps the value to its recording with `value_rec.emplace`, which keeps the **first** one: when the last part finishes the submission, the key is already there and nothing changes. So `wait_recorded` (the game's fence waits in `wait()`, and the exposure readback wait) returns once the first ~120 draws are done. With no VRAM mirror of the buffer pools (integrated GPUs), the later parts read guest memory in place, and the game has by then written the next frame's uniforms, vertices and exposure meter into it. The HUD is drawn late in the frame, so it gets the wrong transforms most often; the stale exposure readback gives the whole-screen flashes. myylkn found the same root cause on a Radeon 8060S (NotRexed#997, with a patch in a comment). This PR is a variant of that patch: - `find` and an update in place rather than `value_rec[v]`, so it builds against `tests/draw_stream_visibility_test.cpp`, which models `value_rec` as a `std::multimap`. - It keeps registering `r.value` when the recording does not signal (the parts split off before the last), raised to the latest seq. A wait on a stopped submission still finds its recording, instead of falling back to the timeline wait, which forces stopped work past its event waits. ## Change - `gpu/vk/vk_replay.cpp`, `submit_recording`: each value maps to the latest recording that carries it. - `tests/test_value_rec_last_part.py`: runs the statements cut from `submit_recording`, for a submission split in three parts and for a stopped submission finished in a later recording. It fails on `main` (`value 10 -> recording 1 (expected 3)`, `value 11 -> recording 4 (expected 5)`) and passes with the change. - `docs/tested-configurations.md`: the three runs below. ## Tested ROG Ally X (Ryzen Z1 Extreme, RADV PHOENIX, Mesa 26.2.3), Linux (Anatase, Fedora-based), counting one-frame luminance spikes in screen recordings (a frame brighter or darker than both its neighbours): | v2.0.13, Hyrule Field | One-frame spikes | 15/255 or more | `GPU waits in 5 s` | |---|---|---|---| | without the change (69 s) | 104 | 15 | 4–109 sync waits, 31–232 ms | | with the change (42 s) | 3 | 0 (the largest is a sword hit) | 103–137 sync waits, 1.4–1.8 s | Then about a day and a half of play with the change, rebuilt for v2.0.14 and v2.1.0 (the renderer alone, `RX_VK_ONLY=1`): no flashes or stray HUD seen in Hyrule Field, Death Mountain Trail and Goron City, in Desktop Mode and in Steam's Gaming Mode. A 57 s Gaming Mode recording on v2.0.14 had no one-frame spikes at all. **It costs frame rate on this device.** The FPS overlay in the field read roughly 45–70 before and 38–49 after (two recordings, not a controlled benchmark): part of the old frame rate was the game running ahead of the GPU. The waits now last as long as the GPU work they stand for. `python -I -B tests/run_all.py` with the Vulkan headers: the same results on a clean checkout of `main` with and without the change, apart from the new test, which passes. `test_draw_stream_visibility`, `test_capture_slot_ownership` and `test_parallel_rb_refresh`, which compile `submit_recording`, pass. **Not tested:** Windows, NVIDIA and discrete AMD cards (the early wait is there too, but their VRAM mirror is synced at the submit, which should hide it; not measured), `RX_VK_PARALLEL` segments (off by default; the same mapping applies to them), and the Steam Deck. ## Related - Part of NotRexed#997 (AMD handhelds and APUs tracker). - NotRexed#1795 copies uniform buffers on GPUs without mirrors for the same symptoms. The two can coexist: that one keeps a late write from reaching the GPU; this one makes the fence waits cover the work they stand for. An A/B of NotRexed#1795 with this change in and `RX_VK_UB_SNAPSHOT=0` would show how much of its effect was this wait. - NotRexed#2125 (merged) lets stopped work finish before a CPU readback. Complementary: the readback's wait that follows goes through `value_rec`, which this corrects. ## Screenshots Before (v2.0.13, Hyrule Field and Castle Town; each strip is the frame before, the broken frame, the frame after): <img width="1920" height="335" alt="before-1" src="https://github.com/user-attachments/assets/4d3ee32e-8575-462e-b972-b3bb16ec2f3e" /> <img width="1920" height="335" alt="before-2" src="https://github.com/user-attachments/assets/2231f72b-3394-46b0-b67d-647674712a29" /> <img width="1920" height="335" alt="before-3" src="https://github.com/user-attachments/assets/d2d645a7-bf2a-41c3-979a-19b2a36562ff" /> <img width="1920" height="335" alt="before-4" src="https://github.com/user-attachments/assets/366157ae-41e2-48a4-b4cd-7fc96f817eaa" /> After (v2.0.13 with the change, Hyrule Field, fights included; the last strip is the largest one-frame spike left, a sword hit): <img width="1920" height="333" alt="after-1" src="https://github.com/user-attachments/assets/9f025025-1034-47ab-bfe9-0ea1476db075" /> <img width="1920" height="333" alt="after-2-largest-spike-is-a-sword-hit" src="https://github.com/user-attachments/assets/e7e26d16-ba97-4982-bed0-670306352113" /> The analysis and the change were made with AI assistance (Claude); the testing and recordings are mine. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: hckrman101 <239247847+hckrman101@users.noreply.github.com> (cherry picked from commit e0dbf40)

…neration off on first start; upstream NotRexed#2419 NotRexed#2420 NotRexed#2439 A Steam Deck ran ~20 fps short of CPU (the GPU waited on 0.4 ms a frame): short sleeps no longer spin in yield loops there (HOS_SHORT_SLEEP_TIMER decides), and made frames no longer take GPU work out of its power budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C7fdTPXbwDRohew9qanvbp

A dGPU has its own fast memory and a slower bus to system RAM; stock mirrors every guest pool into VRAM and keeps both copies coherent with captures, shadows and write-backs. At 4K that coherence is CPU-bound: the write-back diff compares ~3.1 GB per 600 submits and mirror syncs move ~137k dirty pages, for 35.6 fps where 1080p runs 60 (issue NotRexed#2099). Under RX_VK_DGPU=1 a pool's first binding makes no mirror (MIRROR_REFUSED, the state iGPU runs use today): reads take the lock-free TLS path and GPU reads/writes land in the imported guest memory, coherent by construction - no mirror sync, no write-back. Every 600-submit window the heaviest guest-memory-read pool with >=1 MB of reads and not write-dominated runs a measured trial (120 frames direct -> 12 settle, held until the mirror is really on -> 120 mirrored, on the recordings' own timestamps); the mirror stays only when the win is both >10% (RX_VK_DGPU_GAIN) and >0.25 ms (RX_VK_DGPU_FLOOR) on present-to-present frame time, so the verdict sees the mirror's CPU cost and its PCIe benefit alike. Trials whose windows did not measure the same workload (zero binds, a >2x bind shift, a mirror that never came) abort with no verdict and no skip line; a pool that aborts three times sits out the session, and iGPUs run no trials. A lost verdict persists to hookshot_mirror_skip.txt beside the game's own state files, keyed to the device and driver (a foreign or header-less file is ignored and re-trialled). RX_VK_DGPU_TRIALS=0 is pure direct. Trials need serial replay (documented); under RX_VK_PARALLEL the policy runs trial-less. Unset = stock exactly: the added counters and logging are gated on the switch. Refused pools read uniform buffers as submitted (NotRexed#1795's per-pool form: the snapshot engages for mirror-less pools at the read, trial-won pools keep in-place mirror binds), so the artifact class the issue documents on mirror-less GPUs is covered under the policy too. Measured on RX 9070 XT / RADV, 4K, Uncapped frame cap, warp-normalized adjacent pairs (Hyrule Field): 70.2->78.0, 71.0->77.0, 59.6->76.6 fps (+9.7% median), draw parity 705 vs 709, GPU 9.2 ms of a ~14 ms frame. Mirror-sync dirty pages 136,872 -> 0; write-back diff 3.06-3.13 GB -> 330 MB or less per 600 submits. Review round 2 (external): RX_VK_DGPU_TRIALS=0's predicate was inverted and now matches the docs (=0 is pure direct); the window's read counter moved off a shared map into the pool (dgpu_read, both direct paths); Settle holds until the mirror is actually live; zero-bind baselines and >2x bind drift between the direct and mirrored windows abort the trial un-blacklisted; keep_line refuses signed and trailing-garbage lines. Review round 3 (hckrman101, Joey-Magnus): refused pools read UBs as submitted (NotRexed#1795 per-pool, live-smoked at 1.26M copies fault-free); verdicts decide on present-to-present frame time instead of GPU-only time (the mirror's CPU and upload-buffer cost is what the frame pays); the policy runs trial-less under RX_VK_PARALLEL and its fast-path counters are parallel-safe; no trials on iGPUs and a 3-abort cap; the skip list carries a device+driver header and ignores foreign files; the test driver uses {cxx}. Follow-up fix: the drift check counted served volume, so a working mirror's read collapse voided 40+ trials a run and trials never converged; binds now count once at the serving path and the drift rule compares the scene's pool set, with a three-no-verdict cap. Rebased onto main's env_on switch rule (NotRexed#2042) and the split-submission fence work (NotRexed#2419); the trials' functions sit outside the write-back observation's compile-span check (NotRexed#2131) and the GCN mirror-less default (NotRexed#2343) composes unchanged (on GCN the switch runs trial-less direct plus the UB snapshot). Co-authored-by: GLM-5.3-Flash <glm-5.3-flash@opencode.noreply.github.com>

A dGPU has its own fast memory and a slower bus to system RAM; stock mirrors every guest pool into VRAM and keeps both copies coherent with captures, shadows and write-backs. At 4K that coherence is CPU-bound: the write-back diff compares ~3.1 GB per 600 submits and mirror syncs move ~137k dirty pages, for 35.6 fps where 1080p runs 60 (issue NotRexed#2099). Under RX_VK_DGPU=1 a pool's first binding makes no mirror (MIRROR_REFUSED, the state iGPU runs use today): reads take the lock-free TLS path and GPU reads/writes land in the imported guest memory, coherent by construction - no mirror sync, no write-back. Every 600-submit window the heaviest guest-memory-read pool with >=1 MB of reads and not write-dominated runs a measured trial (120 frames direct -> 12 settle, held until the mirror is really on -> 120 mirrored, on the recordings' own timestamps); the mirror stays only when the win is both >10% (RX_VK_DGPU_GAIN) and >0.25 ms (RX_VK_DGPU_FLOOR) on present-to-present frame time, so the verdict sees the mirror's CPU cost and its PCIe benefit alike. Trials whose windows did not measure the same workload (zero binds, a >2x bind shift, a mirror that never came) abort with no verdict and no skip line; a pool that aborts three times sits out the session, and iGPUs run no trials. A lost verdict persists to hookshot_mirror_skip.txt beside the game's own state files, keyed to the device and driver (a foreign or header-less file is ignored and re-trialled). RX_VK_DGPU_TRIALS=0 is pure direct. Trials need serial replay (documented); under RX_VK_PARALLEL the policy runs trial-less. Unset = stock exactly: the added counters and logging are gated on the switch. Refused pools read uniform buffers as submitted (NotRexed#1795's per-pool form: the snapshot engages for mirror-less pools at the read, trial-won pools keep in-place mirror binds), so the artifact class the issue documents on mirror-less GPUs is covered under the policy too. Measured on RX 9070 XT / RADV, 4K, Uncapped frame cap, warp-normalized adjacent pairs (Hyrule Field): 70.2->78.0, 71.0->77.0, 59.6->76.6 fps (+9.7% median), draw parity 705 vs 709, GPU 9.2 ms of a ~14 ms frame. Mirror-sync dirty pages 136,872 -> 0; write-back diff 3.06-3.13 GB -> 330 MB or less per 600 submits. Review round 2 (external): RX_VK_DGPU_TRIALS=0's predicate was inverted and now matches the docs (=0 is pure direct); the window's read counter moved off a shared map into the pool (dgpu_read, both direct paths); Settle holds until the mirror is actually live; zero-bind baselines and >2x bind drift between the direct and mirrored windows abort the trial un-blacklisted; keep_line refuses signed and trailing-garbage lines. Review round 3 (hckrman101, Joey-Magnus): refused pools read UBs as submitted (NotRexed#1795 per-pool, live-smoked at 1.26M copies fault-free); verdicts decide on present-to-present frame time instead of GPU-only time (the mirror's CPU and upload-buffer cost is what the frame pays); the policy runs trial-less under RX_VK_PARALLEL and its fast-path counters are parallel-safe; no trials on iGPUs and a 3-abort cap; the skip list carries a device+driver header and ignores foreign files; the test driver uses {cxx}. Follow-up fix: the drift check counted served volume, so a working mirror's read collapse voided 40+ trials a run and trials never converged; binds now count once at the serving path and the drift rule compares the scene's pool set, with a three-no-verdict cap. Rebased onto main's env_on switch rule (NotRexed#2042) and the split-submission fence work (NotRexed#2419); the trials' functions sit outside the write-back observation's compile-span check (NotRexed#2131) and the GCN mirror-less default (NotRexed#2343) composes unchanged (on GCN the switch runs trial-less direct plus the UB snapshot). After the presenter-thread refactor (NotRexed#2325) the present-to-present tick lives in present_finish, which every made present passes exactly once in either threading mode, with a headless twin so offscreen runs stay measurable. Co-authored-by: GLM-5.3-Flash <glm-5.3-flash@opencode.noreply.github.com>

… no VRAM mirror With no VRAM mirror (mirror_on() off) a draw's uniform buffers are bound in the guest pools, and the GPU reads them when it runs the frame: by then the game has written the next frame's per-object matrices over them. On Intel Core Ultra iGPUs the Kokiri depth prepass drew the ground off screen and the fog pass painted sky over it (the white veil, NotRexed#999). Still there on main after NotRexed#2419. RX_VK_UB_SNAPSHOT (copy the uniforms into the stream as submitted) is now on by default on Intel when the pools are not mirrored. Other unmirrored GPUs (AMD APUs, GCN since NotRexed#2343, RX_VK_MIRROR=0) keep it off until measured. RX_VK_UB_SNAPSHOT=0/1 overrides. Assisted-by: Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A dGPU has its own fast memory and a slower bus to system RAM; stock mirrors every guest pool into VRAM and keeps both copies coherent with captures, shadows and write-backs. At 4K that coherence is CPU-bound: the write-back diff compares ~3.1 GB per 600 submits and mirror syncs move ~137k dirty pages, for 35.6 fps where 1080p runs 60 (issue NotRexed#2099). Under RX_VK_DGPU=1 a pool's first binding makes no mirror (MIRROR_REFUSED, the state iGPU runs use today): reads take the lock-free TLS path and GPU reads/writes land in the imported guest memory, coherent by construction - no mirror sync, no write-back. Every 600-submit window the heaviest guest-memory-read pool with >=1 MB of reads and not write-dominated runs a measured trial (120 frames direct -> 12 settle, held until the mirror is really on -> 120 mirrored, on the recordings' own timestamps); the mirror stays only when the win is both >10% (RX_VK_DGPU_GAIN) and >0.25 ms (RX_VK_DGPU_FLOOR) on present-to-present frame time, so the verdict sees the mirror's CPU cost and its PCIe benefit alike. Trials whose windows did not measure the same workload (zero binds, a >2x bind shift, a mirror that never came) abort with no verdict and no skip line; a pool that aborts three times sits out the session, and iGPUs run no trials. A lost verdict persists to hookshot_mirror_skip.txt beside the game's own state files, keyed to the device and driver (a foreign or header-less file is ignored and re-trialled). RX_VK_DGPU_TRIALS=0 is pure direct. Trials need serial replay (documented); under RX_VK_PARALLEL the policy runs trial-less. Unset = stock exactly: the added counters and logging are gated on the switch. Refused pools read uniform buffers as submitted (NotRexed#1795's per-pool form: the snapshot engages for mirror-less pools at the read, trial-won pools keep in-place mirror binds), so the artifact class the issue documents on mirror-less GPUs is covered under the policy too. Measured on RX 9070 XT / RADV, 4K, Uncapped frame cap, warp-normalized adjacent pairs (Hyrule Field): 70.2->78.0, 71.0->77.0, 59.6->76.6 fps (+9.7% median), draw parity 705 vs 709, GPU 9.2 ms of a ~14 ms frame. Mirror-sync dirty pages 136,872 -> 0; write-back diff 3.06-3.13 GB -> 330 MB or less per 600 submits. Review round 2 (external): RX_VK_DGPU_TRIALS=0's predicate was inverted and now matches the docs (=0 is pure direct); the window's read counter moved off a shared map into the pool (dgpu_read, both direct paths); Settle holds until the mirror is actually live; zero-bind baselines and >2x bind drift between the direct and mirrored windows abort the trial un-blacklisted; keep_line refuses signed and trailing-garbage lines. Review round 3 (hckrman101, Joey-Magnus): refused pools read UBs as submitted (NotRexed#1795 per-pool, live-smoked at 1.26M copies fault-free); verdicts decide on present-to-present frame time instead of GPU-only time (the mirror's CPU and upload-buffer cost is what the frame pays); the policy runs trial-less under RX_VK_PARALLEL and its fast-path counters are parallel-safe; no trials on iGPUs and a 3-abort cap; the skip list carries a device+driver header and ignores foreign files; the test driver uses {cxx}. Follow-up fix: the drift check counted served volume, so a working mirror's read collapse voided 40+ trials a run and trials never converged; binds now count once at the serving path and the drift rule compares the scene's pool set, with a three-no-verdict cap. The set rule still voided every trial in a streaming scene, so anchors now match by guest range (a recycled pool is the same anchor), anchor at 4x the floor, void only when a substantial fraction of the scene moves, and the candidate floor rose to 256 MB of reads a window (the verdict floor is unreachable below it - a trial there is churn by construction). Rebased onto main's env_on switch rule (NotRexed#2042) and the split-submission fence work (NotRexed#2419); the trials' functions sit outside the write-back observation's compile-span check (NotRexed#2131) and the GCN mirror-less default (NotRexed#2343) composes unchanged (on GCN the switch runs trial-less direct plus the UB snapshot). After the presenter-thread refactor (NotRexed#2325) the present-to-present tick lives in present_finish, which every made present passes exactly once in either threading mode, with a headless twin so offscreen runs stay measurable. Co-authored-by: GLM-5.3-Flash <glm-5.3-flash@opencode.noreply.github.com>

… them (NotRexed#2602) Session 6 found the synced build's lost quarter of its frame rate in the game's readbacks of GPU results: upstream NotRexed#2419's wait for a submission's last part and NotRexed#2125's settle under sync_order_mu. A fix written here (kept as backup/readback-by-part-mine) took Kokiri Forest from 39 to 53.5 fps on the i7-11700 and RTX 3080; upstream merged the same fix, more complete, as NotRexed#2602, which this fork takes with its next sync instead. Area: docs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… no VRAM mirror (#2464) This is my first contribution to Hookshot, so I am happy to change anything about the approach, the style or the scope. I did the investigation and the change with the help of AI, as the commit trailers say. I checked the results on my own machine, described below. On Intel Core Ultra iGPUs, Kokiri Forest is covered by a white veil: the ground is missing and the sky/fog is painted over it. The mechanism is not Intel-specific (any GPU whose buffer pools have no VRAM mirror can read uniform buffers stale), but the new default is limited to Intel for now. ## Why With no VRAM mirror (`mirror_on()` off: integrated GPUs, GCN since #2343, `RX_VK_MIRROR=0`), a draw's uniform buffers are bound straight into the imported guest pools. The GPU reads them when it runs the frame. By then the game has already written the next frame's per-object matrices over the ones that were submitted. In Kokiri, the depth prepass (generic program `ffb53ed102dd9908`) drew the terrain with the wrong `c4` matrices, so the terrain landed off screen. The depth there stayed at far (1.0). The fog/atmosphere composite (`1a7482c0ec9991ea`) then treats every pixel above 0.9999 as sky and painted fog over the ground. The forward pass draws the ground colour correctly, which is why only the fog layer looked wrong. On a discrete GPU the mirror is copied at submission time, so it never sees the rewrite. ## What changed `RX_VK_UB_SNAPSHOT` already copies a uniform buffer into the stream as submitted (`memory_read_as_submitted`). It is now on by default on Intel when the pools are not mirrored (`vendorID == 0x8086 && !pools_mirrored()`). Other GPUs without a mirror (AMD APUs, Steam Deck, GCN, `RX_VK_MIRROR=0`) keep it off until someone measures the cost there. `RX_VK_UB_SNAPSHOT=0` turns it off; `=1` forces it on anywhere. ## How it was found - `RX_VK_DUMP_EACH=ffb53ed102dd9908` + `RX_VK_DUMP_ARGS` read the depth back after each prepass draw. 551 of 558 draws added no depth at all, including 12k-24k-index terrain meshes drawn while the depth was still empty. - A CPU projection of the same draws, using the logged vertices and `c3`/`c4`, put 454 of them on screen with depth ~0.99. - Writing depth with compare ALWAYS, no cull and depth clamp changed nothing. Neither did copying the vertex and index buffers into the stream as submitted. Copying the uniform buffers did: the prepass depth went from ~45,000 far samples (of 57,600) to ~1,000, and the veil went away. ## Testing - **Machine:** Lenovo laptop (type 83DJ). - **CPU:** Intel Core Ultra 7 155H (Meteor Lake). - **GPU:** integrated Intel Arc Graphics (8 Xe-cores), no dedicated GPU; video memory is shared with system RAM. - **RAM:** 32 GB LPDDR5x-7467. - **OS:** Windows 11 Pro, build 26300 (Insider). - **Driver:** Intel 32.0.101.9175, installed by hand (the version from #741). - **Game settings:** output resolution setting 3 (`HOS_OUTPUT_RES=3`), `RX_VK_FIFO=1`, `RX_VK_PREPARE_BACKGROUND=1`. - **Build:** v2.1.0 with this change; then current `main` (3e32949, with #2419) built from source with this branch rebased on it. - **Current main without this change:** the veil is still there (#2419 does not cover it). Numbers and screenshots in the comment below. - **Kokiri, from the house to the village:** - before: white veil over the ground in every wide view; - after: ground, paths, deck and ivy correct, light fog as intended; - frame rate unchanged, ~8.5-11 fps in Kokiri at the same settings. - No AMD or discrete GPU tested; the default there is unchanged. Related: #999 (Intel tracker); report and workaround: #999 (comment) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

… them (NotRexed#2602) Session 6 found the synced build's lost quarter of its frame rate in the game's readbacks of GPU results: upstream NotRexed#2419's wait for a submission's last part and NotRexed#2125's settle under sync_order_mu. A fix written here (kept as backup/readback-by-part-mine) took Kokiri Forest from 39 to 53.5 fps on the i7-11700 and RTX 3080; upstream merged the same fix, more complete, as NotRexed#2602, which this fork takes with its next sync instead. Area: docs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…nd no game thread waits behind a present for nothing Hyrule Field lost about 15 fps between v2.0.11-588 (ba86aee) and main (RTX 3090, 2560x1440, uncapped: 79 -> 64-71 fps, CPU 39 -> 64-73 ms a frame). One of the game's job threads was blocked 5.5 ms a frame in a CPU read of GPU results (0.1 ms before), and the game's other job threads polled with 20 us sleeps meanwhile (1.9 cores in sched_yield against 0.3). The read is of the culling results the first part of the scene's submission writes; that submission then stops at an event wait and is finished later, on the presenting thread. Three things held the read up: - since NotRexed#2419 a submission's value maps to its last part's recording, so the read waited for the GPU's whole submission. Each part writes back the pages the GPU wrote in it, at its own end: the page table now keeps the part with the value, and the read waits for the newest part of its pages. A page written in place (no mirror) keeps the wait for its whole submission, whose later parts read and write guest memory itself (issue 997). RX_VK_READBACK_PART=0: the last part, as before. - since NotRexed#2125 the read took sync_order_mu whenever a queue was stopped, behind the presenting thread's whole replay and its present. A queue can be stopped at a met wait with nobody to let it on only after a CPU signal; the read looks at the stopped queues when the CPU has signalled since the last look (2-9 of 3000 readbacks, all of them before). RX_VK_READBACK_SETTLE_ALWAYS=1: whenever anything is stopped, as before. - a wait whose recording was already done, and the host's copy bookkeeping (rx_nvn_replayed, under its own lock), took dev.present_mu to flush held submits, which exist only with batched submits. A present holds that mutex 4 ms a frame. The flushes look at an atomic first; a wait returns at once when the recordings' timeline is already there. Hyrule Field, same session, fix / base / main: 78.6 / 81.3 / 63.1 and 78.7 / 73.9 / 65.9 fps; CPU 46.7 / 38.3 / 71.0 and 45.1 / 40.0 / 69.5 ms a frame. Readbacks that waited: ~1350 of 3000 at 0.14 ms (main 172 at 5.5 ms, base ~1300 at 0.13 ms). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

A submission longer than RX_VK_CHUNK draws, or one stopped at an event wait and finished later, is recorded in several recordings that all carry its value. submit_recording mapped the value with emplace, which kept the FIRST part's seq, so wait_recorded (the game's fence waits and the exposure readback wait) returned once the first ~120 draws were done. With no VRAM mirror (integrated GPUs) the later parts read guest memory in place, and the game had written the next frame's uniforms, vertices and exposure meter into it: a frame of the HUD stretched across the screen, grass white, whole-screen flashes (issue 997). Each value now maps to the latest recording that carries it, raised to its seq in place; a stopped submission's value is still registered by the parts before the one that signals. The rule moved to gpu/vk/rec_value.h so tests/test_rec_value.cpp can check it without a device. Ported from upstream e0dbf40 (NotRexed#2419).

… them (NotRexed#2602) Session 6 found the synced build's lost quarter of its frame rate in the game's readbacks of GPU results: upstream NotRexed#2419's wait for a submission's last part and NotRexed#2125's settle under sync_order_mu. A fix written here (kept as backup/readback-by-part-mine) took Kokiri Forest from 39 to 53.5 fps on the i7-11700 and RTX 3080; upstream merged the same fix, more complete, as NotRexed#2602, which this fork takes with its next sync instead. Area: docs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

willcrisis and others added 2 commits November 11, 2026 15:04

Merge remote-tracking branch 'origin/main' into fix/value-rec-last-part

c16d691

# Conflicts: # docs/tested-configurations.md

willcrisis marked this pull request as ready for review November 11, 2026 18:30

Joey-Magnus approved these changes Nov 11, 2026

View reviewed changes

Merge branch 'main' into fix/value-rec-last-part

9306a97

hckrman101 merged commit e0dbf40 into NotRexed:main Nov 11, 2026

github-actions Bot mentioned this pull request Nov 11, 2026

chore(main): release 2.2.0-rc.1 #2071

Closed

This was referenced Nov 11, 2026

fix(vk): snapshot uniform buffers on Intel when the buffer pools have no VRAM mirror #2464

Merged

perf(vk): eliminate rec_timeline write-back stalls and capture deadlock #2458

Merged

sheepbun mentioned this pull request Nov 11, 2026

chore(fork): sync main with upstream's at 3e329497 sheepbun/Bunny-Hood#3

Merged

Joey-Magnus mentioned this pull request Nov 11, 2026

fix(vk): HUD and lighting artifacts on GPUs without buffer mirrors #1795

Open

blessedplummer mentioned this pull request Nov 12, 2026

perf(vk): a PC-like dGPU policy (RX_VK_DGPU=1) - direct guest-memory binds, mirrors only where a measured trial proves them #2338

Open

SadTransGirl mentioned this pull request Nov 12, 2026

feat(vk): RX_VK_WAIT_FIRST_PART=1, an A/B of #2419's fence-wait cost … #2587

Closed

futondestroyer mentioned this pull request Nov 12, 2026

[Tracking] Six-hour main rebuilds and in-game performance benchmarks #2595

Open

Efraim2704 mentioned this pull request Nov 12, 2026

[Tracking] Low FPS with low GPU usage (CPU-bound), FPS drops in Castle Town / Hyrule Field, shader stutter #995

Open

Zarigis mentioned this pull request Nov 12, 2026

#2419 (fence wait covers every part of a split submission) costs about 9 fps on NVIDIA at 4K: 60 → 51 in Hyrule Field #2628

Open

Brett-D mentioned this pull request Nov 12, 2026

Merge upstream main through c544828f (restore RX_VK_MESH_NOCULL) omgitsgela/Hookshot-SteamDeck#4

Open

Brett-D mentioned this pull request Nov 12, 2026

docs(tested): RTX 5090 on Windows, Kokiri Forest: main against v2.1.0, 49f75dc8 and the last main before #2602 #2840

Open

Navigation Menu

Repository navigation

fix(vk): a fence wait covers every part of a split submission, not only the first - #2419

fix(vk): a fence wait covers every part of a split submission, not only the first#2419
hckrman101 merged 3 commits into
NotRexed:mainfrom
willcrisis:fix/value-rec-last-part

willcrisis commented Nov 11, 2026 •

edited

Loading

Uh oh!

Joey-Magnus left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

willcrisis commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Cause

Change

Tested

Related

Screenshots

Uh oh!

Joey-Magnus left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

willcrisis commented Nov 11, 2026 •

edited

Loading