Repository navigation
perf(build): lifted chunks of ~4 MB, by guest address, besides chunks.txt's boundaries - #2890
Open
jordanella wants to merge 1 commit into
Open
jordanella wants to merge 1 commit into
jordanella wants to merge 1 commit into
Conversation
….txt's boundaries Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dazemc
added a commit
to dazemc/Hookshot-rs
that referenced
this pull request
Nov 12, 2026
Port the five upstream build-performance changes to builder/rust (timing.rs untouched): - NotRexed#2775: the lifted code (chunks and hot pieces) compiles with -Xclang -disable-lifetime-markers. Measured on an authored 6 MiB lifted-style chunk (clang 23, -O1): peak RSS 428.6/428.5/427.2 MB without the flag -> 393.6/390.8/392.7 MB with it (~8.4% lower); wall time 36.3-47.7 s -> 35.9-48.8 s, neutral within the noise. The memory gate can therefore fit more compiles at once. - NotRexed#2706: the back end's own files compile with -MD and an explicit -MF (the object is written to a .part first), and their object keys use only the headers their depfile names under gpu/vk, gpu/render, gpu/compat and suyu's textures. Eden/sirit keep their own header blob. An object keyed by the last depfile is stamped depfile-pending and cached only by the key its own new depfile gives; a build that stops before the re-keying recompiles it. A header edit now rekeys the files whose depfile names it, not the whole back end (vk_present_rect.h is included by two built sources, not 32). - NotRexed#2893: link_key/run_link skip a link whose command, compiler, object stamps and other named files (inside options too, such as the symbol order) are the last successful link's; a failed link or an output of another size forgets the key. The host executable, hookshot_game and the Vulkan library all link through it. compile_commands.json is written beside the objects at <work>/build/compile_commands.json, merged from the game, host and Vulkan stages, with the clangd schema (directory/file/output/arguments). - NotRexed#2890: the current 6 MB UNIT_BYTES split is by compile cost, not by a stable guest address, so a size change can move its cuts. The lift now also starts a chunk at the first function at or after each 0x40000 of guest address (SUB_STRIDE), giving ~3.7 MB chunks under UNIT_BYTES; a function found later moves at most one boundary, and only the recorded chunks.txt boundaries are saved, never the sub-boundaries. - NotRexed#2748: audited, no port needed. The Rust lift already emits the missed functions in native threads (HOOKSHOT_LIFT_JOBS, default 16) on Windows and Linux, sharing the lifter read-only; unlike nx2/lift.py's Windows path it needs no worker processes. Tests: cargo test --workspace passes (290 lib tests plus the integration suites), including new checks for the lifted flags, depfile parsing/keys and re-keying, link skipping, the compile_commands schema, and chunk boundaries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A small update that changes a few of the game's functions still recompiles every lifted chunk that holds one, and a chunk is ~24 MB of C: a typical one takes ~227 s of compiler time, and the few heavy ones far more. Splitting the lifted C into ~4 MB pieces by guest address makes that ~41 s, with no cost in play or in a full build. Part of the dev-loop proposal's work on small updates (#2636).
Change
nx2/chunks.py:image/chunks.txt's stable boundaries,sub_startsadds one everySUB_STRIDE(0x40000) of guest address, at the first function at or after it. They are guest addresses, so they stay put when a function grows: a change moves no boundary but its own chunk's.chunks.txtkeeps only the boundaries it had (the sub-boundaries are worked out again each lift from the same functions), so dropping this change later is clean.split_big,split_hotandbuild.pymatch three-digit chunk names.The first build after this compiles every lifted chunk once (all boundaries are new): for a player, that update takes about as long as a full build, once. Updates after it recompile only the ~4 MB pieces that changed.
Measured
Windows 11, Ryzen 9 3900X:
mainagainst 32:30, at the same total compiler time (25,700 against 25,500 s) over 698 files instead of 486. The smaller pieces spread better over the cores.In the game
No change in play: the 4 MB-chunk game against the 24 MB one, Kokiri Forest from a save (the game logic bounds its frame on the CPU there), RTX 3080 (NVIDIA 621.14), Windows 11, two interleaved pairs of 30 samples: median 29.6 / 30.0 fps against 31.0 / 30.0. No errors in either.
Testing
tests/test_sub_chunks.py(new): the first function past each stride; a long function spans strides without a boundary inside it; a function found since moves one boundary at most; chunks end at them as well as at the saved boundaries;writeuses them and saves onlychunks.txt's; the stride makes chunks of a few MB. (The 1000-chunk guard isn't tested: the game's code makes ~330 pieces.)run_all.py, rebased on today'smain(with perf(lift): emit the lifted C in several processes on Windows too #2748's spawned emit), Windows: 8 pass;test_spawn_emitfails the same way onmainhere.See #2636.
🤖 Generated with Claude Code