Skip to content

perf(build): compile the lifted code without lifetime markers - #2775

Open
corentinbonjour wants to merge 1 commit into
NotRexed:mainfrom
corentinbonjour:perf/no-lifetime-markers
Open

corentinbonjour wants to merge 1 commit into
NotRexed:mainfrom
corentinbonjour:perf/no-lifetime-markers

Conversation

@corentinbonjour

Copy link
Copy Markdown
Contributor

Building the game is limited by memory on most PCs: the memory gate starts a compiler only when its peak fits, and a 25 MB lifted chunk takes about 1.5 GB. Leaving out clang's lifetime markers cuts every lifted compile's peak by about 10%, with no measurable run-time cost on lifted third-party code.

What changed

  • nx2/build.py flags(): adds -Xclang -disable-lifetime-markers. The lifted chunks, the hot pieces (hot_fl is built from fl) and the files built with them (a64_table.c, a64_rt.c, the shim) all get it. The module docstring lists it with the other lifted-code flags.
  • tests/test_lifted_flags.py (new): checks that the pair reaches the compiler at -O1 and -O2 and that the level is unchanged. When --cxx is clang, it also compiles a scoped local with the lifted flags and checks that the IR has no llvm.lifetime call. A reference compile without the pair must have one, so the check is not empty. If a future clang drops the cc1 option, the test fails here before a build breaks.
  • mem_need() is unchanged. On the synthetic chunk it estimates 1736 MB against a measured 1476 MB before this change and 1324 MB after, so it is still an upper bound. Its coefficients come from game chunks, and I have not re-measured those.

Measurements

Synthetic input: third-party AArch64 code lifted with this branch's lifter into a 25.4 MB chunk (code_000.c) and a 6.0 MB chunk (code_001.c). No game files. Compiler: llvm-mingw clang 23 from the builder (v2.0.14), MinGW target, the exact build.flags() list with and without the pair. I ran 5 repetitions with the two variants side by side in each, so both saw the same load. Each compile ran in its own Windows job: CPU is the whole process tree, and peak is the largest working set of any process in it. Values are medians with [min..max].

case metric without the flag with the flag change
25 MB chunk, -O1 peak working set 1476 MB [1475..1476] 1324 MB [1324..1325] -10.3%
25 MB chunk, -O1 peak commit 1452 MB 1298 MB -10.6%
25 MB chunk, -O1 CPU 203.6 s [171.6..217.8] 200.4 s [167.2..214.7] -1.6% (every pair -3 to -4 s)
6 MB chunk, -O2 peak working set 367 MB [367..367] 330 MB [330..331] -10.1%
6 MB chunk, -O2 CPU 44.6 s [30.9..49.0] 42.8 s [30.9..48.0] -4.1%

An earlier run of the same comparison, on a hot piece of 8.7 MB compiled at -O2 with a real profile (from an instrumented build of the same synthetic code), gave the same -10% peak, and the profile still applied: 2066 branch weights and 118 entry counts with and without the flag. Lifted zstd decompression and Opus encode/decode timed in that harness (QueryThreadCycleTime, 12 to 30 interleaved rounds) did not change beyond the noise.

Code side effects on the synthetic chunks:

  • .text grows 1.1% at -O1 and 0.4% at -O2.
  • Without the markers LLVM can no longer share stack slots between scopes. At -O1, 237 of 1655 functions with a frame get a bigger one. The median stays at 40 B, the p99 goes from 328 B to 2.2 KB, and the largest from 440 B to 3.7 KB, still under a page, so no stack probe is added. At -O2, 10 of 779 grow.
  • Guest threads ask for 64 MB stacks, so this is far from the limit.

How it was tested

  • python -I -B tests/test_lifted_flags.py --cxx clang++ passes here (MSVC and MinGW targets). On an untouched main it fails on the first check.
  • python -I -B tests/run_all.py with llvm-mingw on PATH and RX_CPM set, here and on an untouched origin/main worktree: the same 13 checks fail on both, all from tools missing on this PC. Three more (test_pipeline_cache_save, test_direct_buffer_writes, test_oot_wide_save_picture) failed only on the branch's run, from compile timeouts while a lift was running alongside. All three pass when re-run alone.
  • test_mem_gate* and test_mem_limit pass.

Notes

  • The flags are part of every object key, so the first build after updating compiles the game code again, once.
  • -Xclang is clang-only. The lifted code is always built with clang (toolchain.find("clang")) on both platforms.

Pass -Xclang -disable-lifetime-markers with the lifted-code flags (chunks and
hot pieces). Each compiler peaks about 10% lower (1476 -> 1324 MB on a 25 MB
-O1 chunk, 367 -> 330 MB at -O2), so the memory gate fits more compiles at
once. Compile CPU drops 2-4%. mem_need stays as it is, an upper bound.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@Joey-Magnus Joey-Magnus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I merged this into current main and ran tests/test_lifted_flags.py --cxx clang++ with clang 21 and tests/test_build_times.py. Both pass, and the IR check finds llvm.lifetime in the reference compile and none with the flags, so the cc1 option works on an older clang too. The change itself is one flag at nx2/build.py:610. Two things for the maintainers to weigh rather than bugs:

  • Every object key includes the flags, so every player's first build after this lands recompiles all the lifted game code once. On low-RAM PCs that is the slow, memory-gated part. The benefit is about 10% lower peak per compiler from then on, which also lets the memory gate start more compilers.
  • The measurements are on synthetic third-party code lifted with the same lifter, not on Ocarina's chunks. That applies to both the peak memory and the "runs as fast" part (zstd and Opus timings). It is probably representative, but a peak-working-set comparison on one real game chunk, and a frame-time check in the game, would back the claim with the code it ships for. Bigger stack frames (p99 from 328 B to 2.2 KB) are fine on the 64 MB guest stacks.

dazemc added a commit to dazemc/Hookshot-rs that referenced this pull request Nov 12, 2026
Port the five upstream build-performance changes to builder/rust (timing.rs
untouched):

- NotRexed#2775: the lifted code (chunks and hot pieces) compiles with
  -Xclang -disable-lifetime-markers. Measured on an authored 6 MiB
  lifted-style chunk (clang 23, -O1): peak RSS 428.6/428.5/427.2 MB without
  the flag -> 393.6/390.8/392.7 MB with it (~8.4% lower); wall time
  36.3-47.7 s -> 35.9-48.8 s, neutral within the noise. The memory gate can
  therefore fit more compiles at once.

- NotRexed#2706: the back end's own files compile with -MD and an explicit -MF (the
  object is written to a .part first), and their object keys use only the
  headers their depfile names under gpu/vk, gpu/render, gpu/compat and suyu's
  textures. Eden/sirit keep their own header blob. An object keyed by the
  last depfile is stamped depfile-pending and cached only by the key its own
  new depfile gives; a build that stops before the re-keying recompiles it.
  A header edit now rekeys the files whose depfile names it, not the whole
  back end (vk_present_rect.h is included by two built sources, not 32).

- NotRexed#2893: link_key/run_link skip a link whose command, compiler, object stamps
  and other named files (inside options too, such as the symbol order) are
  the last successful link's; a failed link or an output of another size
  forgets the key. The host executable, hookshot_game and the Vulkan library
  all link through it. compile_commands.json is written beside the objects
  at <work>/build/compile_commands.json, merged from the game, host and
  Vulkan stages, with the clangd schema (directory/file/output/arguments).

- NotRexed#2890: the current 6 MB UNIT_BYTES split is by compile cost, not by a
  stable guest address, so a size change can move its cuts. The lift now also
  starts a chunk at the first function at or after each 0x40000 of guest
  address (SUB_STRIDE), giving ~3.7 MB chunks under UNIT_BYTES; a function
  found later moves at most one boundary, and only the recorded chunks.txt
  boundaries are saved, never the sub-boundaries.

- NotRexed#2748: audited, no port needed. The Rust lift already emits the missed
  functions in native threads (HOOKSHOT_LIFT_JOBS, default 16) on Windows and
  Linux, sharing the lifter read-only; unlike nx2/lift.py's Windows path it
  needs no worker processes.

Tests: cargo test --workspace passes (290 lib tests plus the integration
suites), including new checks for the lifted flags, depfile parsing/keys and
re-keying, link skipping, the compile_commands schema, and chunk boundaries.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants