Skip to content

fix(vk): tree-shadow flicker without the TAAU network: the shadow mask evened out over frames (rebase of #477) - #1581

Merged
hckrman101 merged 3 commits into
NotRexed:mainfrom
rakefire-impala:fix/tree-shadow-flicker-history
Nov 11, 2026
Merged

hckrman101 merged 3 commits into
NotRexed:mainfrom
rakefire-impala:fix/tree-shadow-flicker-history

Conversation

@rakefire-impala

@rakefire-impala rakefire-impala commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Rebase of #477 by @imbahzor onto current main, as asked when it was closed ("please rebase on current main and reopen"). The commit is theirs, cherry-picked with -x; the only changes are the conflict resolution and the push-descriptor mark, which now goes through main's descriptors_disturbed() (see Change).

Problem

Over sunlit ground and along shadow edges near trees, shadows flicker constantly in thin streaks, even with the camera still. This happens on GPUs where the gameplay TAAU network is left out ("a network left out on this card" in the log, for example an RX 5500M), so the bilinear stand-in runs instead.

The tree-shadow pass 3427a1dbefd5e02c is a full-screen draw. It rebuilds a normal from an ImageGather of the neighbouring depths and offsets its shadow-map lookup by that normal. On grass blades and leaf edges, those neighbours flip between the blade and the ground behind it as the camera shifts by a sub-pixel each frame, so the pixels flip between lit and shadowed. On the console, the TAAU network averages that away.

Change

  • Mask history. Right after that draw, its mask (G and B of an RGBA8 target at the scene's size, MIN blended) is blended with its own history in a small compute pass, shaders/shadow_stab.comp: two history images used in turn, then G and B copied back.
    • RX_VK_MASK_BLEND=<w>: the weight, default 0.85; 0 turns it off.
    • RX_VK_MASK_RADIUS=<r>: limits the history to a (2r+1)² neighbourhood. Default 0 (no limit), because a 3×3 limit kept nothing.
  • Push-descriptor cache. The new pass pushes descriptors of its own layout, so it calls descriptors_disturbed() right after the push. Without that, the next dispatch with the cached layout skipped its push and read the pass's bindings, which caused a device lost after a save load in vk: tree-shadow flicker evened out; push-descriptor cache stale after the stand-ins' pushes #477's testing. vk: tree-shadow flicker evened out; push-descriptor cache stale after the stand-ins' pushes #477 also marked movie_upscale and taau_stand_in; main now does that itself.

Tests

tests/test_tree_shadow_mask.py (new) checks that the blend follows the tree-shadow pass only while RX_VK_MASK_BLEND is above 0, needs an RGBA8 storage mask and push descriptors, runs outside the render pass, and that every push of the stand-ins' or the mask's own layout reaches descriptors_disturbed() before its function returns. It fails with any one of those three calls removed.

Tested

imbahzor and others added 3 commits November 10, 2026 17:00
…e after our own pushes

Tree shadows (OoT's 3427a1dbefd5e02c) flickered in fast thin streaks over sunlit ground and shadow edges while the
camera stood still. The pass offsets each pixel's shadow-map lookup by a normal it reconstructs from a gather of the
neighbouring depths; on grass blades and leaf edges those neighbours flip between the blade and the ground behind it
with the sub-pixel camera shift, so pixels flip between lit and shadowed every frame (the console's TAAU averages that
away). Right after that draw, its mask (G and B of an RGBA8 target, MIN blended) is blended with its own history
(shaders/shadow_stab.comp). RX_VK_MASK_BLEND=<w> (default 0.85, 0: off), RX_VK_MASK_RADIUS=<r> (history bound,
default 0: none; a 3x3 bound kept nothing, the streaks move as blocks).

The TAAU and movie stand-ins push descriptors of their own layout without telling bind_descriptors' cache, so the
next dispatch of the cached layout could skip its push and read the stand-in's bindings (found as a device lost after
a save load once another compute pass was inserted after the network).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[rebased on main 9447e1d: main now marks the cache stale after the stand-ins (descriptors_disturbed), so only the
new pass does it here, through that helper]
(cherry picked from commit 29ab07f)
…our own pushes

Checks that the blend follows the tree-shadow pass (3427a1dbefd5e02c) only
while RX_VK_MASK_BLEND is above 0, needs an RGBA8 storage mask and push
descriptors and runs outside the render pass, and that every push of the
stand-ins' or the mask's own layout marks bind_descriptors' cache stale
(fails with one of the three marks removed).
…story

The header still described the history as bounded by the 3x3 neighbourhood; the default is radius 0 (unbounded), as the
comment at the bound and RX_VK_MASK_RADIUS's description say. Comment only: the SPIR-V is unchanged.
@rakefire-impala
rakefire-impala force-pushed the fix/tree-shadow-flicker-history branch from c10a3b4 to 52ea5bf Compare November 10, 2026 23:00
@rakefire-impala

Copy link
Copy Markdown
Contributor Author

Rebased on main 9447e1d. The only conflict was the line just before the new guard in draw(): dlss_skin_check is skin_check on main now.

One more change came with the rebase. Main now calls descriptors_disturbed() after movie_upscale and taau_stand_in push their own descriptors, so this PR's marks there are gone, and the new pass uses that helper too. The test checks that all three pushes reach it and fails with any one removed.

I also fixed your first nit: the header comment of shadow_stab.comp now says the history is unbounded by default. It's a comment-only change, so the SPIR-V is unchanged. I left the other two nits alone so the code stays as you reviewed it. #1637 and #1776 are both in main already, and this merges on top of them.

@hckrman101
hckrman101 merged commit 746fa11 into NotRexed:main Nov 11, 2026
9 of 10 checks passed
TechHammerBR pushed a commit to TechHammerBR/RecompZel that referenced this pull request Nov 11, 2026
…ce lost near Hyrule Castle) (NotRexed#2374)

**Release blocker for 2.1.1.** The owner's test of main crashed near
Hyrule Castle: device lost after a ~2.5 s GPU hang on the mesh draw
`b0b3b40244c71ec7`. In the checkpoints, the draw that follows it is
OoT's tree-shadow pass `3427a1dbefd5e02c`, the one NotRexed#1581's mask blend
(`mask_stabilize`) runs after.

| build, the owner's castle save, native mesh | result |
|---|---|
| v2.1.0 (before NotRexed#1581) | 48,960 presents, no crash |
| main `49f75dc8` | device lost at ~30,000, **twice** |
| main, mesh through the compute path | 49,560, no crash (flashes) |
| main, `RX_VK_MASK_BLEND=0` | **51,600, no crash** |

Two of last night's route runs (`sun_nodefer`, `c8_on`) died the same
way, on the same program.

This makes the blend opt-in (`RX_VK_MASK_BLEND=<weight>`; the default
was 0.85). Without the TAAU network, the tree-shadow streaks it evened
out come back until the hang is found. `tests/test_tree_shadow_mask.py`
is updated (passes).

cc the author of NotRexed#1581: the hang is on the draw before your pass, so it
may be the mask image's history or barrier rather than the blend itself.
Logs are available.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
NotRexed pushed a commit that referenced this pull request Nov 12, 2026
…ols the GPU writes (#2154)

> **Update (2026-11-11):** a second commit widens the fix to save
reloads and logs the mirrors still given back. The pause fix below is
tested in game; the second commit is not yet (see "Save reloads").

## Problem

Pause the game, leave it in the background for 1–2 minutes (another
window or the Start menu in front), then come back and resume: for a few
seconds the picture has cyan patches. On an RX 5500M (RDNA1, 4 GB,
Windows 11, AMD driver) this happened on every resume.

While the game is paused, most of its memory pools are not bound. After
`kIdleEpochs` (600) submissions an unbound pool's VRAM mirror is idle,
and `mirrors_give_back` gives it back once it has been idle past
`kLongIdle` (6000 submissions) or when a wanted mirror does not fit. In
one such pause the running count in the log went from 35 to 51 (`51
mirrors given back (1136 MB in all)`). At the pool's next binding,
`mirror_make` makes a new mirror from guest memory (plus the CPU's newer
writes from the captures).

Most of what the GPU writes into a mirror is copied back to guest
memory: a written binding queues a write-back. But some of it is never
copied back, by design, and is noted only as a hazard by
`memory_note_gpu_write`: what a shader writes through a bare entry's
guessed range, the pointer table's range or a wide binding. (Copying the
whole guessed range back overwrote the CPU's own data next to it.) That
data was in the old mirror alone. The new mirror starts without it, and
the GPU goes on from stale data until it has rebuilt it, which is the
few seconds of cyan.

## Change

- `Pool::gpu_only`: set by `memory_note_gpu_write` on every imported
pool its range reaches.
- `mirrors_give_back` leaves an idle `gpu_only` pool's mirror alone. Its
other idle mirrors go as before.
- A pool that takes others in (`pool_lookup`'s merge) keeps their mark:
their mirrors' contents come with them.
- The give-back's log line also says how many MB of idle mirrors are
kept this way.
- Second commit: `Pool::gpu_written`, set by `gpu_buffer_bind` for a
written binding of the mirror, carried over a merge like `gpu_only`, and
the idle give-back keeps those mirrors too.
- Second commit: every mirror still given back is logged (the first
200), with why (retired, retired and not inherited in time, idle, or
idle for room), how many submissions since its sync, and the pool's
marks.

A pool that is refused a mirror for the budget meanwhile is read in
system memory, which is slower but not wrong. Retired pools are
unchanged: their GPU contents already go to the pool that took them in
(`inherit_pending`).

## Tested

RX 5500M, Windows 11, AMD driver, main `92907f7` with #1581 and #2125,
same settings throughout. Each pause was 1–2 minutes with the game in
the background:

| | cyan after resume | mirrors given back during the run |
|---|---|---|
| main (`92907f7` + #1581 + #2125) | every resume (tried at Auto), for a
few seconds | 16 in one pause |
| `RX_VK_MIRROR_KEEP=1` (no give-back at all) | none, one pause at Auto
and one at 33% Render resolution | 0 |
| this PR | none, one pause at Auto and one at 33% | 48 in the session;
112 MB of idle mirrors kept at the last count |

A very short black flash right on resume is still there at both
resolutions. At Auto it has been there since the first builds on this
card, including the `RX_VK_MIRROR_KEEP=1` run, so it looks separate.

## Save reloads

With the first commit, cyan came back after reloading a save within a
run (Hyrule Castle, Hyrule Field), in play and over the Start menu, with
no pause and no `not imported` pools in the log. (That run also had
`RX_VK_FIFO=1`, which turned out not to matter.) Against
`RX_VK_MIRROR_KEEP=1`, which never gives a mirror back:

| | run 1 | run 2 |
|---|---|---|
| first commit only | cyan after each reload (castle, field) | — |
| `RX_VK_MIRROR_KEEP=1` | three reloads, no cyan over the Start menu |
fewer flashes, some on the third reload |

So mirrors given back explain most of the cyan after a reload, but not
all of it. The first commit keeps only pools the GPU wrote past the
write-back. The pools whose mirrors still went are the ones the GPU
writes through a normal binding, which the second commit keeps too.

**Not tested in game yet:** the second commit. I haven't played a build
with it, so I don't know yet whether it matches `RX_VK_MIRROR_KEEP=1` or
what it costs in VRAM. Its log line is there to answer both, and to show
which mirrors are still given back before any flash that's left.

## Tests

`tests/test_mirror_gpu_only.py` (new) reads the source and checks four
things: `memory_note_gpu_write` marks the pools it reaches,
`gpu_buffer_bind` marks a pool written through its mirror, the idle
give-back leaves marked pools, and a merge keeps both marks. It fails on
current main and passes with this change. `nx2/test_import_windows.py`
builds `pool_lookup` against a `Pool` of its own, which now has both
fields.

`tests/run_all.py` on main `4d68bc83` with this branch merged: 311
passed. The 6 that fail are the same 6 that fail on plain main on this
PC.

Related: #1005 (cyan and UI flashes after focus loss), #2125.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants