Skip to content

fix(build): the lift key is taken again only if the lift's other inputs did not change - #2686

Merged
hckrman101 merged 2 commits into
NotRexed:mainfrom
jordanella:fix/lift-key-inputs-unchanged
Nov 12, 2026
Merged

hckrman101 merged 2 commits into
NotRexed:mainfrom
jordanella:fix/lift-key-inputs-unchanged

Conversation

@jordanella

Copy link
Copy Markdown
Contributor

#2674 can certify stale lifted C: if a translator file is edited while a lift runs, the key taken after the lift describes the new file, but the C came from the old one, and the next build skips a lift it needs. mooki000 found and reproduced this on #2674 just before it merged; this closes it.

Change

This is mooki000's suggested shape: refresh the self-produced inputs only after confirming the rest are unchanged.

Testing

  • tests/test_lift_cleanup_error.py: a new case where an input changes during the lift writes the pre-lift key, and the key taken after doesn't validate. Against main's make_game.py it fails; with this change it passes. The existing cases still see lift → key taken again → key written.
  • tests/test_lift_cache.py:
    • The source order: the inputs snapshot is taken before nx2/lift.py runs, and the check comes before the key is taken again.
    • planning=False ignores changes to hot_pieces.txt and chunks.txt, but follows the game's code and the hot list.
    • Fails against main's make_game.py, passes with this change.
  • Every check that reads make_game.py (52), on Windows, passes apart from test_cpp_preflight, test_linux_update and test_prepare_stall, which fail the same way on a clean main here.
  • Not run: mooki000's gist fixture, which this machine's tool permissions wouldn't let me run. Its guarded variant is the same check this PR adds; a run with --variant pointed at this branch's commit would confirm it.

🤖 Generated with Claude Code

@mooki000

Copy link
Copy Markdown
Contributor

The run you asked for on #2674, of the fixture with its two commit ids set to current main and this head (Linux, Python 3.12, no game, no compiler):

a translator file edited during the lift only the lift's own planning files change
main c544828f (has #2674) fails: the old output is certified, the next build would skip passes
this head a29018fa passes: not certified, the next build lifts again (1 lift) passes: the next build skips (0 lifts), "the game was translated with these tools before"

So the window is closed and #2674's gain is kept. Your own two test files pass here on this head (tests/test_lift_cache.py, tests/test_lift_cleanup_error.py), and with main's nx2/make_game.py put under them tests/test_lift_cache.py and tests/test_lift_cleanup_error.py fail, as you say.

One case neither version nor the code before #2674 sees, for the record and not for this pull request: a file changed and changed back during one lift (a branch switched and switched back while it runs) compares equal before and after.

Not tested: Windows, and a real lift (the fixture stands in for the lift and the split; everything around them is the repository's code).

@SimonKimDev SimonKimDev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small window left in the check: the full key is taken after the comparison, so a translator file edited between the second lift_key(..., planning=False) and lift_key(exefs, work, env) still stamps C from the old translator with the new key.

Taking the full key first closes it:

after = lift_key(exefs, work, env)
if lift_key(exefs, work, env, planning=False) == inputs:
    key = after

An edit before after is caught by the comparison; an edit after it doesn't matter, since after already hashed the old file.

@jordanella

Copy link
Copy Markdown
Contributor Author

@SimonKimDev right, thanks: the order left exactly that gap. 54e2652 takes the full key first and keeps it only if the planning=False comparison after it holds, as you wrote; test_lift_cache.py now checks that order, and fails on the previous commit.

@mooki000 thank you for running the fixture on both sides; that's the confirmation I couldn't get here. Noted on a file changed and changed back within one lift (a branch switched and back): no snapshot of hashes sees that. It's out of this PR, as you say.

…nputs did not change

NotRexed#2674 stores the key taken after the lift, so that the files the lift writes for the next one (image/hot_pieces.txt,
image/chunks.txt) are in it. Taken whole, that key also covered a translator file edited while the lift ran, and
certified C made from the old file under the new file's key: the next build skipped a lift it needed (mooki000's
fixture on NotRexed#2674).

translate() now takes lift_key(planning=False), everything but those two files, as the lift starts and again after it,
and takes the key again only if that is unchanged. Otherwise the key from before stays, and the next build translates
again, as before NotRexed#2674. tests/test_lift_cleanup_error.py adds the case of an input changed during the lift, and
tests/test_lift_cache.py checks the order and what planning=False leaves out and keeps.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mooki000

Copy link
Copy Markdown
Contributor

The same fixture on 54e26526: unchanged from a29018fa. A translator file edited during the lift is not certified and the next build lifts again (1 lift); with only the planning files changing, the next build skips (0 lifts); main c544828f still fails the first case.

The key was taken after lift_key(planning=False) was compared, so a translator file edited between the two would still
stamp C from the old file with the new key (SimonKimDev on NotRexed#2686). It is now taken first and kept only when the
comparison after it holds: an edit before it is caught by the comparison, and one after it is in no key that is kept.
tests/test_lift_cache.py checks that order, and the cleanup test's changed-input case sees the key taken and not kept.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dazemc added a commit to dazemc/Hookshot-rs that referenced this pull request Nov 12, 2026
Upstream NotRexed/Hookshot fixed four builder problems the fork shares, on
the path its Flutter builder and Rust engine use. Ported semantically,
with the Rust engine's own layout and failure handling:

- e7f5581: scripts/oot_saves.py finds Lichii's own data folder
  (~/.local/share/lichii, %APPDATA%\lichii) before Eden's, where a Lichii
  before build 150 still keeps its saves. `list` groups programs by the
  folder import and export use; the fork's `saves` list for the Flutter
  launcher follows the same rule, so Lichii is not offered from Eden's.
- 6c529a2: a Btrfs folder has a device number no disk has, so the hard
  drive warning could not tell; nx2/disk.py and builder/rust/src/disk.rs
  now read /proc/self/mountinfo for the deepest mount and every disk of a
  multi-disk Btrfs (/sys/fs/btrfs/<id>/devices).
- c544828 (+ NotRexed#2686, NotRexed#2675): the lift key is taken again after the lift
  and split_hot, from image/chunks.txt that the first lift writes, so a
  new work folder's second build no longer translates the game again; the
  new key is kept only when nothing else the lift reads changed while it
  ran. A compile that failed only in repository sources keeps the
  translation (a host typo no longer costs a lift); a lifted or generated
  file that failed still forgets it. The Rust driver never had a key but
  had no such distinction: compile failures now follow the same rule, a
  link failure forgets the lift, and a failed host source keeps it.
- a740d67: host_mtune()/host::mtune() detect a Steam Deck APU (0405 or
  0932) or RX_STEAMDECK=1 and add -mtune=znver2 beside -march, in
  nx2/build.py and the Rust driver's game and host compiles; RX_MTUNE
  names a tune. The fork's lift and host builds both get it.

Tests: the upstream ones adapted (test_oot_saves, test_disk_warning,
test_cpu_target, test_lift_cache, test_lift_cleanup_error), new
tests/test_failed_compile_keeps_lift.py, and Rust tests for the Btrfs
mount walk, the lift failure rule and the Zen 2 tune. No game files or
generated game data.

Assisted-by: opencode (DeepSeek V4.1 Flash 1017)

@Joey-Magnus Joey-Magnus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SimonKimDev's ordering point is fixed in 54e2652. after = lift_key(...) is now taken before the planning=False comparison at nx2/make_game.py:2190, and test_lift_cache.py checks that order. mooki000's fixture on the new head confirms both cases: an edit made during the lift is not certified, and a build where only the planning files changed still skips the lift. I merged this into current main. tests/test_lift_cleanup_error.py passes (9 tests, including the new "input changed during the lift" case). tests/test_lift_cache.py has the same single failure here as on main (numpy isn't installed for the lift import check), and all the new checks pass. The fallback is the conservative one: if anything changed during the lift, the pre-lift key stays and the next build lifts again. That costs a few minutes at worst, never a stale build. An edit reverted during the lift (A to B to A) can't be seen by a content hash, but the C would be from A anyway, so that's fine. It merges cleanly with #2693, which also edits make_game.py.

@hckrman101
hckrman101 merged commit 331efae into NotRexed:main Nov 12, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants