Repository navigation
Stop the replay and exit when the GPU device is lost - #1291
Merged
Merged
Conversation
After VK_ERROR_DEVICE_LOST the replay only wrote the checkpoint report and went on. No submission completed again, so no capture or staging arena was let go, and system memory filled until the window was closed (issue 1275: captures kept 25 -> 6527, staging 256 -> 1408 MB). The four places that see the loss now call device_lost(): the report once (std::call_once, so a second thread that sees the loss waits for it instead of returning at once), then replay_fatal, exit code 3. tests/test_device_lost_exit.py checks that every VK_ERROR_DEVICE_LOST in vk_replay.cpp goes to device_lost(), and runs device_lost() from five threads with a slow report: one report, and the process ends only after it. Refs NotRexed#1275 The LLM council (several models answer the same question independently, then rank each other's answers anonymously) was used to define the approach and the solution: which problem to take on, how to fix it and what to avoid. It saw a summary of the issues, not the code. The change and its test were written with Claude Code, then reviewed and tested by me. Assisted-by: Claude Opus 5.5 Assisted-by: gpt-oss-120b (LLM council) Assisted-by: Qwen3.8-27B (LLM council) Assisted-by: Gemini 3.5 Flash-Lite (LLM council) Assisted-by: dots-3-note-preview (LLM council) Assisted-by: Nemotron 3.5 Lightning (LLM council) Assisted-by: Nemotron 3 Ultra 550B (LLM council) Assisted-by: Nemotron 3.5 Super VL (LLM council) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Windows with a console the host queues its log lines and a background thread writes them; replay_fatal ended the process with _Exit right after logging, which could drop the queue. With device_lost() the queue holds the whole checkpoint report, so it now goes first: MkwVkHost gains log_flush (hos_log_flush; API version 10), and replay_fatal calls it before _Exit. Linux is unchanged (its log is not queued). tests/test_device_lost_exit.py checks that replay_fatal flushes before it exits and that rx_backend.c passes hos_log_flush; the parallel-init-order fixture gains the host it now reads. Review feedback from hckrman101 on NotRexed#1291. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Thanks for the careful read. Fixed in fb879db: |
zeusward1-a11y
added a commit
to zeusward1-a11y/Hookshot
that referenced
this pull request
Nov 10, 2026
…ice lost ends the game too PR NotRexed#1130's update: on an RTX 3070 the parts made the Hyrule Castle load hitch ~1.2 s longer (2.1 s against 0.9 s) with no driver reset to avoid, so bc_part_budget's default is 16 M texels on AMD and 0 (all at once, as before the parts) elsewhere. RX_VK_BC_PART_MTEXELS=16 brings them back on any GPU. tests/test_bc_parts.py checks the default. PR NotRexed#1291 took four VK_ERROR_DEVICE_LOST checks to device_lost(); this branch has a fifth, in submit_segment_recording (the parallel replay), which now goes there as well (tests/test_device_lost_exit.py refuses a stray one). Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
After
VK_ERROR_DEVICE_LOSTthe replay wrote the checkpoint report and keptrunning. Nothing it submitted completed again, so captures and staging were
never retired: in #1275
captures keptwent from 25 to 6527 and staging from256 MB to 1408 MB, until system RAM ran out with a black window.
The four
VK_ERROR_DEVICE_LOSTchecks ingpu/vk/vk_replay.cppnow call a newdevice_lost(), which writes the report once throughstd::call_onceand thenends the process through the existing
replay_fatal(exit code 3). Withcall_once, a second thread that sees the loss at the same time (the #1275 logshows several) waits for the report instead of ending the process before it is
written.
This does not fix the GPU hang itself in #1275, only what happens after it.
Tests
tests/test_device_lost_exit.py(new, game-free): everyVK_ERROR_DEVICE_LOSTcheck must calldevice_lost();device_lost()is cutinto a fixture and called from five threads with a slow report, which must be
written once before the exit. It fails on the old code path and on a
check-once flag without
call_once.tests/test_parallel_init_order.py: its fixture gained adevice_loststub(the segment-submit block it cuts now calls it); passes.
changed function was compiled, in the fixture). Linux, g++.
Question
Do you want the process to stay alive after a device lost in some debugging
setup (e.g. with
RX_VK_FAULT_INFO)? If so, I can put the exit behind anopt-out variable.
Refs #1275
AI assistance
Assisted-by: Claude Opus 5.5
Assisted-by: gpt-oss-120b (LLM council)
Assisted-by: Qwen3.8-27B (LLM council)
Assisted-by: Gemini 3.5 Flash-Lite (LLM council)
Assisted-by: dots-3-note-preview (LLM council)
Assisted-by: Nemotron 3.5 Lightning (LLM council)
Assisted-by: Nemotron 3 Ultra 550B (LLM council)
Assisted-by: Nemotron 3.5 Super VL (LLM council)
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
The LLM council (several models answer the same question independently, then
rank each other's answers anonymously) was used to define the approach and the
solution: which problem to take on, how to fix it and what to avoid. It saw a
summary of the issues, not the code. The change and its test were written with
Claude Code, then reviewed and tested by me.