Skip to content

perf(build): lifted chunks of ~4 MB, by guest address, besides chunks.txt's boundaries - #2890

Open
jordanella wants to merge 1 commit into
NotRexed:mainfrom
jordanella:perf/smaller-chunks
Open

jordanella wants to merge 1 commit into
NotRexed:mainfrom
jordanella:perf/smaller-chunks

Conversation

@jordanella

Copy link
Copy Markdown
Contributor

A small update that changes a few of the game's functions still recompiles every lifted chunk that holds one, and a chunk is ~24 MB of C: a typical one takes ~227 s of compiler time, and the few heavy ones far more. Splitting the lifted C into ~4 MB pieces by guest address makes that ~41 s, with no cost in play or in a full build. Part of the dev-loop proposal's work on small updates (#2636).

Change

nx2/chunks.py:

  • Sub-boundaries: besides image/chunks.txt's stable boundaries, sub_starts adds one every SUB_STRIDE (0x40000) of guest address, at the first function at or after it. They are guest addresses, so they stay put when a function grows: a change moves no boundary but its own chunk's.
  • Not saved: chunks.txt keeps only the boundaries it had (the sub-boundaries are worked out again each lift from the same functions), so dropping this change later is clean.
  • A guard: more than 1000 chunks stops the lift with a message, since split_big, split_hot and build.py match three-digit chunk names.

The first build after this compiles every lifted chunk once (all boundaries are new): for a player, that update takes about as long as a full build, once. Updates after it recompile only the ~4 MB pieces that changed.

Measured

Windows 11, Ryzen 9 3900X:

  • One chunk's compile: a typical 24 MB chunk ~227 s of compiler time against ~41 s for a 4 MB piece. The heavy-function cluster (old chunks 021-023) still takes 300-900 s per piece.
  • A full build with my local build changes together (this, the lift key's environment, link skips and others), each in a fresh player-like work folder: 36:17 on main against 32:30, at the same total compiler time (25,700 against 25,500 s) over 698 files instead of 486. The smaller pieces spread better over the cores.
  • A two-PR update (fix(vi): the time of day passes again above a 60 fps cap #2093 and feat: add separate button mappings for the ocarina notes #1930, the same setup): 6:06 against 4:36; compiling 2:16 against 1:00.

In the game

No change in play: the 4 MB-chunk game against the 24 MB one, Kokiri Forest from a save (the game logic bounds its frame on the CPU there), RTX 3080 (NVIDIA 621.14), Windows 11, two interleaved pairs of 30 samples: median 29.6 / 30.0 fps against 31.0 / 30.0. No errors in either.

Testing

  • tests/test_sub_chunks.py (new): the first function past each stride; a long function spans strides without a boundary inside it; a function found since moves one boundary at most; chunks end at them as well as at the saved boundaries; write uses them and saves only chunks.txt's; the stride makes chunks of a few MB. (The 1000-chunk guard isn't tested: the game's code makes ~330 pieces.)
  • Every check that reads the changed files (9), through run_all.py, rebased on today's main (with perf(lift): emit the lifted C in several processes on Windows too #2748's spawned emit), Windows: 8 pass; test_spawn_emit fails the same way on main here.
  • Not tested: Linux.

See #2636.

🤖 Generated with Claude Code

….txt's boundaries

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dazemc added a commit to dazemc/Hookshot-rs that referenced this pull request Nov 12, 2026
Port the five upstream build-performance changes to builder/rust (timing.rs
untouched):

- NotRexed#2775: the lifted code (chunks and hot pieces) compiles with
  -Xclang -disable-lifetime-markers. Measured on an authored 6 MiB
  lifted-style chunk (clang 23, -O1): peak RSS 428.6/428.5/427.2 MB without
  the flag -> 393.6/390.8/392.7 MB with it (~8.4% lower); wall time
  36.3-47.7 s -> 35.9-48.8 s, neutral within the noise. The memory gate can
  therefore fit more compiles at once.

- NotRexed#2706: the back end's own files compile with -MD and an explicit -MF (the
  object is written to a .part first), and their object keys use only the
  headers their depfile names under gpu/vk, gpu/render, gpu/compat and suyu's
  textures. Eden/sirit keep their own header blob. An object keyed by the
  last depfile is stamped depfile-pending and cached only by the key its own
  new depfile gives; a build that stops before the re-keying recompiles it.
  A header edit now rekeys the files whose depfile names it, not the whole
  back end (vk_present_rect.h is included by two built sources, not 32).

- NotRexed#2893: link_key/run_link skip a link whose command, compiler, object stamps
  and other named files (inside options too, such as the symbol order) are
  the last successful link's; a failed link or an output of another size
  forgets the key. The host executable, hookshot_game and the Vulkan library
  all link through it. compile_commands.json is written beside the objects
  at <work>/build/compile_commands.json, merged from the game, host and
  Vulkan stages, with the clangd schema (directory/file/output/arguments).

- NotRexed#2890: the current 6 MB UNIT_BYTES split is by compile cost, not by a
  stable guest address, so a size change can move its cuts. The lift now also
  starts a chunk at the first function at or after each 0x40000 of guest
  address (SUB_STRIDE), giving ~3.7 MB chunks under UNIT_BYTES; a function
  found later moves at most one boundary, and only the recorded chunks.txt
  boundaries are saved, never the sub-boundaries.

- NotRexed#2748: audited, no port needed. The Rust lift already emits the missed
  functions in native threads (HOOKSHOT_LIFT_JOBS, default 16) on Windows and
  Linux, sharing the lifter read-only; unlike nx2/lift.py's Windows path it
  needs no worker processes.

Tests: cargo test --workspace passes (290 lib tests plus the integration
suites), including new checks for the lifted flags, depfile parsing/keys and
re-keying, link skipping, the compile_commands schema, and chunk boundaries.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant