Skip to content

Stop the replay and exit when the GPU device is lost - #1291

Merged
hckrman101 merged 2 commits into
NotRexed:mainfrom
berelels:fix/exit-on-device-lost
Nov 9, 2026
Merged

hckrman101 merged 2 commits into
NotRexed:mainfrom
berelels:fix/exit-on-device-lost

Conversation

@berelels

@berelels berelels commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

What

After VK_ERROR_DEVICE_LOST the replay wrote the checkpoint report and kept
running. Nothing it submitted completed again, so captures and staging were
never retired: in #1275 captures kept went from 25 to 6527 and staging from
256 MB to 1408 MB, until system RAM ran out with a black window.

The four VK_ERROR_DEVICE_LOST checks in gpu/vk/vk_replay.cpp now call a new
device_lost(), which writes the report once through std::call_once and then
ends the process through the existing replay_fatal (exit code 3). With
call_once, a second thread that sees the loss at the same time (the #1275 log
shows several) waits for the report instead of ending the process before it is
written.

This does not fix the GPU hang itself in #1275, only what happens after it.

Tests

  • tests/test_device_lost_exit.py (new, game-free): every
    VK_ERROR_DEVICE_LOST check must call device_lost(); device_lost() is cut
    into a fixture and called from five threads with a slow report, which must be
    written once before the exit. It fails on the old code path and on a
    check-once flag without call_once.
  • tests/test_parallel_init_order.py: its fixture gained a device_lost stub
    (the segment-submit block it cuts now calls it); passes.
  • Not tested: a real device lost on hardware, and a full build (only the
    changed function was compiled, in the fixture). Linux, g++.

Question

Do you want the process to stay alive after a device lost in some debugging
setup (e.g. with RX_VK_FAULT_INFO)? If so, I can put the exit behind an
opt-out variable.

Refs #1275

AI assistance

Assisted-by: Claude Opus 5.5
Assisted-by: gpt-oss-120b (LLM council)
Assisted-by: Qwen3.8-27B (LLM council)
Assisted-by: Gemini 3.5 Flash-Lite (LLM council)
Assisted-by: dots-3-note-preview (LLM council)
Assisted-by: Nemotron 3.5 Lightning (LLM council)
Assisted-by: Nemotron 3 Ultra 550B (LLM council)
Assisted-by: Nemotron 3.5 Super VL (LLM council)
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

The LLM council (several models answer the same question independently, then
rank each other's answers anonymously) was used to define the approach and the
solution: which problem to take on, how to fix it and what to avoid. It saw a
summary of the issues, not the code. The change and its test were written with
Claude Code, then reviewed and tested by me.

berelels and others added 2 commits November 9, 2026 17:57
After VK_ERROR_DEVICE_LOST the replay only wrote the checkpoint report and
went on. No submission completed again, so no capture or staging arena was
let go, and system memory filled until the window was closed (issue 1275:
captures kept 25 -> 6527, staging 256 -> 1408 MB).

The four places that see the loss now call device_lost(): the report once
(std::call_once, so a second thread that sees the loss waits for it instead
of returning at once), then replay_fatal, exit code 3.

tests/test_device_lost_exit.py checks that every VK_ERROR_DEVICE_LOST in
vk_replay.cpp goes to device_lost(), and runs device_lost() from five
threads with a slow report: one report, and the process ends only after it.

Refs NotRexed#1275

The LLM council (several models answer the same question independently,
then rank each other's answers anonymously) was used to define the
approach and the solution: which problem to take on, how to fix it and
what to avoid. It saw a summary of the issues, not the code. The change
and its test were written with Claude Code, then reviewed and tested by me.

Assisted-by: Claude Opus 5.5
Assisted-by: gpt-oss-120b (LLM council)
Assisted-by: Qwen3.8-27B (LLM council)
Assisted-by: Gemini 3.5 Flash-Lite (LLM council)
Assisted-by: dots-3-note-preview (LLM council)
Assisted-by: Nemotron 3.5 Lightning (LLM council)
Assisted-by: Nemotron 3 Ultra 550B (LLM council)
Assisted-by: Nemotron 3.5 Super VL (LLM council)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Windows with a console the host queues its log lines and a background
thread writes them; replay_fatal ended the process with _Exit right after
logging, which could drop the queue. With device_lost() the queue holds the
whole checkpoint report, so it now goes first: MkwVkHost gains log_flush
(hos_log_flush; API version 10), and replay_fatal calls it before _Exit.
Linux is unchanged (its log is not queued).

tests/test_device_lost_exit.py checks that replay_fatal flushes before it
exits and that rx_backend.c passes hos_log_flush; the parallel-init-order
fixture gains the host it now reads.

Review feedback from hckrman101 on NotRexed#1291.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hckrman101
hckrman101 merged commit 17f88c1 into NotRexed:main Nov 9, 2026
@berelels

Copy link
Copy Markdown
Contributor Author

Thanks for the careful read. Fixed in fb879db: MkwVkHost gets log_flush (hos_log_flush, API version 10), and replay_fatal calls it before _Exit; test_device_lost_exit.py checks both. On the report hanging with RX_VK_FAULT_INFO: agreed, I left it, since it only happens with that debug variable. Keeping exit-by-default as you suggest.

zeusward1-a11y added a commit to zeusward1-a11y/Hookshot that referenced this pull request Nov 10, 2026
…ice lost ends the game too

PR NotRexed#1130's update: on an RTX 3070 the parts made the Hyrule Castle load hitch ~1.2 s longer (2.1 s against 0.9 s) with no
driver reset to avoid, so bc_part_budget's default is 16 M texels on AMD and 0 (all at once, as before the parts) elsewhere.
RX_VK_BC_PART_MTEXELS=16 brings them back on any GPU. tests/test_bc_parts.py checks the default.

PR NotRexed#1291 took four VK_ERROR_DEVICE_LOST checks to device_lost(); this branch has a fifth, in submit_segment_recording
(the parallel replay), which now goes there as well (tests/test_device_lost_exit.py refuses a stray one).

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant