Skip to content

building.md

Latest commit

 

History

History
234 lines (175 loc) · 13.7 KB

building.md

File metadata and controls

234 lines (175 loc) · 13.7 KB
Raw

Building from source

Windows 10 or later, an x86-64 CPU with AVX2, and a GPU with Vulkan 1.3 and VK_EXT_external_memory_host. Allow about 60 GB for the work folder. Extraction wants about 55 GB free while it runs and keeps about 35 GB afterwards.

Players who only want to play can use the installer instead: it builds the game on their machine without this toolchain.

Toolchain

Versions are the ones the build was verified with; newer patch releases are fine.

Tool Version Install Used for
MSYS2 UCRT64 clang/lld 23.1.3 msys2.org installer to C:\msys64, packages below compiling the generated C and the runtime
Visual Studio Build Tools 2026, C++ workload, Windows SDK 10.0.26100 winget Microsoft.VisualStudio.BuildTools with --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended the linker .NET NativeAOT calls
.NET SDK 10.0 winget Microsoft.DotNet.SDK.10 the shader translator (totk_xlate.exe)
Vulkan SDK 1.4.363 winget KhronosGroup.VulkanSDK validation layers, vulkaninfo
uv 0.13 standalone installer from astral.sh the two Python environments
hactool 1.4.0 @ 1d64a83 .venv\Scripts\python tools\build_hactool.py, see below NCA and update extraction

MSYS2 packages, after pacman -Syuu (run it twice; the first pass updates the core and exits):

pacman -S --needed --noconfirm mingw-w64-ucrt-x86_64-{clang,lld,ninja,gcc,make,pkgconf,sdl3,opus,libvpx,zstd,vulkan-devel,\
vulkan-memory-allocator,vulkan-validation-layers,shaderc,spirv-tools} base-devel git unzip

ffmpeg from MSYS2 is a GPLv3 build and is never linked; the movies go through libvpx.

PATH:

  • Add C:\Program Files (x86)\Microsoft Visual Studio\Installer to the user PATH: NativeAOT needs vswhere.exe, or the publish fails at the link step (MSB3073).
  • Another cmake or ninja earlier on PATH (a 32-bit winlibs MinGW, for example) is picked up by accident. The build scripts call Ninja from C:\msys64\ucrt64\bin themselves; from a shell, put C:\msys64\ucrt64\bin first.

hactool

tools/build_hactool.py fetches SciresM/hactool at 1d64a83 into toolchain\src\hactool, applies the three patches in tools/hactool/ and builds a static toolchain\hactool\hactool.exe with MSYS2 UCRT64 gcc (a few seconds):

  • portable.patch: the parts of MacNativeGames' build_hactool() that run on Windows, the MNG_TITLE_KEYS variable and the two BKTR fixes in nca.c for the subsection table padding and the subsection layout of newer cumulative updates;
  • unicode.patch: a manifest with <activeCodePage>UTF-8</activeCodePage>, so paths under a non-ASCII user profile reach it intact;
  • mbedtls-asm-clobbers.patch: the memory and cc clobbers the bundled mbedtls 2.6.1's x86-64 multiply asm lacks. Without them a clang build (the Linux release) fails every RSA signature check and the update merge stops.

hactool reads -k <prod.keys>; MSYS2 shells set HOME, which hides its default key location, so pass -k there.

Build

From the repository root, in PowerShell. Every step that depends on the game version takes it as <ver>: v143 for 1.4.3, v121 for 1.2.1 or v111 for 1.1.1. The tools default to v121 when --version is left out. Saves load on any of them.

  1. Python environments (once). .venv runs the lifter and the build; .venv-py314 runs the extraction scripts, which need the stdlib compression.zstd of Python 3.14:

    uv venv --python 3.12 .venv
uv pip install --python .venv/Scripts/python.exe -r tools/requirements.txt
uv venv --python 3.14 .venv-py314
uv pip install --python .venv-py314/Scripts/python.exe -r tools/requirements-py314.txt

  2. Copy config.example.toml to config.local.toml and set the paths to your base game, the update and your console's keys file. config.local.toml is ignored by git. The tools read the keys file themselves.

  3. Extract and verify. This writes build\extracted\v0 (1.0.0) and build\extracted\<ver> for each update in updates, identified by its hash; a hash mismatch stops the run:

    .venv-py314\Scripts\python tools\extract.py

  4. Shader translator (once; no game files). This publishes build\xlate\totk_xlate.exe:

    .venv\Scripts\python tools\build_xlate.py

  5. Lift the modules to C, write the RomFS overlay (the frame-rate fix), then compile and link build\<ver>\totk.exe:

    .venv\Scripts\python recomp\gen.py --exefs build\extracted\<ver>\exefs --out generated\<ver> --patches recomp\patches-<ver>.txt
.venv-py314\Scripts\python tools\make_romfs_patches.py --version <ver>
.venv\Scripts\python tools\build_game.py --version <ver>

    Lifting takes about 15 s, a clean compile about 200 s on 24 threads; later builds recompile only what changed. A compiler cache in front of clang shortens it further: with SCCACHE set to an sccache executable, build_game.py writes it into the ninja file ahead of cc, and every lift rewrites the generated chunks with the same contents, so a second build of the same version hits all of their objects:

    $env:SCCACHE = "path\to\sccache.exe"
.venv\Scripts\python tools\build_game.py --version <ver>

    --patches builds in the frame-rate, resolution and other patch sets, switchable at run time. The link copies SDL3.dll, libiconv-2.dll and libopus-0.dll next to the exe. For Xbox button prompts, run make_romfs_patches.py once more with --xbox-prompts: it writes build\<ver>\romfs-patches-xbox (controls.md, "Xbox button prompts"); build_game.py copies it next to totk.exe like romfs-patches.

  6. Prepare the shaders (once per build folder and translator). It needs refs\MacNativeGames (tools\fetch_refs.sh) for the shared shader list:

    .venv-py314\Scripts\python tools\prepare_shaders.py build\<ver> --version <ver>

    Without this step the first launch compiles shaders and pipelines during play and stutters until the caches fill. The AMD driver keeps its pipeline cache per exe path and holds a limited amount: preparing another build folder can evict this one's. totk.exe --warm-shaders rebuilds it from the shader cache.

  7. Run. The game keeps running when its window loses the focus (assume_focus = 1 in totk.ini, the default; the Game tab switches it). With 0 it pauses, as it does on the Switch when Home is pressed.

    build\<ver>\totk.exe

Settings live in totk.ini next to totk.exe (totk.ini.example lists them). Saves go to build\profile for v121 and build\profile-<ver> for other versions (build_game.py --profile changes it).

Portable folder

.venv\Scripts\python tools\package_build.py --version <ver> writes build\<ver>-portable, which runs on another of your machines without MSYS2, Python or .NET. romfs.bin stays out unless you pass --with-romfs; otherwise put it in romfs\ next to the exe or point TOTK_ROMFS at its folder. Run totk.exe --prepare-shaders once in the copied folder. The folder holds code and data derived from your game: for your own machines only.

Linux (in progress)

The runtime builds and runs on x86-64 Linux with a window, input and menu on SDL3 (*_sdl.c) in place of the Win32 files (*_win.c); runtime/platform/linux.c implements platform.h. Status: the selftest and the probes pass; 1.2.1 plays at 60 fps on RADV (RX 7800 XT) and NVIDIA (RTX 3060); 1.4.3 builds, probes and plays on RADV (RX 6600, and the Vega 7 of a Ryzen 5 4600G at about 34 fps). Long play sessions and the Steam Deck itself are not verified yet.

Ocarina 1.1.1 uses the same runtime: the library and the executable link, --selftest and the graphics probes (--probe-gfx-abi, --probe-astc, --probe-vkmem, --probe-window) pass on NVIDIA (RTX 4050 Laptop, Vulkan 1.4), and the shader translator builds for linux-x64 (tested 2026-11-06). In-game play still has to be confirmed with a real copy of the game.

compilar.sh runs the Ocarina steps of this guide (extract, lift, pack, build), with --simular to show the commands and --sem-perguntar to skip the questions:

./compilar.sh --jogo /path/your-game.nup        # or a folder with exefs and romfs
./jogar/jogar-ocarina.sh

On a laptop with an Intel or AMD iGPU next to an NVIDIA GPU, start it with PRIME offload: prime-run ./jogar/jogar-ocarina.sh (CachyOS: prime-run game-performance ./jogar/jogar-ocarina.sh, or prime-run game-performance %command% as the launch options of a Steam shortcut to the script). If the script prints "Driver/library version mismatch", the NVIDIA driver was updated without a restart and Vulkan only sees the iGPU until you restart.

Requirements:

  • Linux 6.7 or later: the write watch uses userfaultfd write protection with PAGEMAP_SCAN. SteamOS 3.8 qualifies.

  • A Vulkan 1.3 driver with VK_EXT_external_memory_host (RADV, NVIDIA).

  • Packages on Arch:

    pacman -S --needed clang lld ninja make pkgconf sdl3 opus dav1d libvpx zstd vulkan-headers vulkan-icd-loader \
    vulkan-validation-layers shaderc spirv-tools dotnet-sdk uv git

    On Debian and Ubuntu the development packages are named libsdl3-dev, libopus-dev, libdav1d-dev, libvpx-dev, libzstd-dev, libvulkan-dev and dotnet-sdk-10.0. ov_codec.c decodes AV1 with dav1d, so the link needs -ldav1d; without it, build_game.py stops at undefined symbol: dav1d_get_picture.

    Vulkan Memory Allocator is a single header (AUR vulkan-memory-allocator). Without it, build_game.py fetches the version the Windows release pins (3.4.0) into toolchain/vma/ on first use and checks its hash.

The steps are the Windows ones with POSIX paths: .venv/bin/python in place of .venv\Scripts\python, and totk in place of totk.exe. build_game.py takes clang, lld and Ninja from PATH (or the ones in <dir>/bin with --toolchain <dir>) and links against the system libraries, with libstdc++ static and an $ORIGIN rpath; build_hactool.py builds hactool with make. tools/prepare_shaders.py reads the shader list from refs/MacNativeGames at origin/HEAD, which must be the pinned commit with its history (the list format changed later).

The probes open a window, so from an SSH session they need the desktop session's display, or they stop at gpu 0 skipped: no graphics+compute+present queue family: on Wayland XDG_RUNTIME_DIR=/run/user/$(id -u) WAYLAND_DISPLAY=wayland-1 SDL_VIDEODRIVER=wayland (the socket name is in $XDG_RUNTIME_DIR), on X11 DISPLAY=:0.

Memory: on Windows the GPU imports guest memory and GetWriteWatch tracks the CPU's writes to it. On Linux, re-protecting imported memory invalidates the driver's userptr buffer on every submit (about 1.3 ms per 16 MiB on amdgpu), and importing pins the pages, which drops their write protection. Each CPU-visible pool therefore gets a host-visible copy (shadow storage, runtime/vk_memory.c): the pages the CPU wrote go in before each submit, and the GPU's writes come back at its fence. In play that costs about 0.2 ms a frame and doubles the memory of those pools (about 1.1 GB). TOTK_SHADOW=1 turns it on under Windows too. The Steam Deck's RADV has VK_EXT_external_memory_host as well: the shadow copies are chosen for the measured cost above, not for a missing extension.

The Deck's GPU (Van Gogh) has no native ASTC, so its textures go through the compute decoder (runtime/vk_astc.c, TOTK_ASTC) as on desktop AMD GPUs without it; frame times measured there include it.

Checks

Before a commit:

  • python tools/check_content.py --staged: nothing game-derived is staged. CI runs the same check; git config core.hooksPath .githooks runs it before each local commit.
  • build\<ver>\totk.exe --selftest: layout, dispatch table and host-function registry, without game files.

The lifter (recomp/) and the CPU headers (runtime/cpu.h, runtime/simd.h) are checked against Unicorn. Every lifter change passes these, all run with .venv:

Test Checks Game files
tests\diff\lift.py instructions sampled from the extracted modules; --coverage lists every word the lifter rejects or traps yes
tests\diff\simd.py the whole FP/SIMD encoding space no
tests\diff\atomic.py exclusive load/store sequences and ARMv8.1 atomics no
tests\diff\func.py whole functions as the generator emits them, loops, jump tables and calls included (calls go to a stub that checks the registers at the call) yes
tests\flags_check.py the function-local flag helpers of runtime\cpu.h against its packed ones no
tests\excl_mt.py the exclusive monitor helpers under contention and ABA, on several threads no
tests\jumptables_check.py recomp/jumptables.py against a Unicorn oracle; must report 0 disagreements, on 1.4.3 too yes

Runtime probes, each a mode of totk.exe (the list is at the top of runtime/main.c): --probe-kernel, --probe-vkexec, --probe-nvn-tex and --probe-shader cover most runtime changes. --probe-shader and the draw part of --probe-vkexec translate shaders of the census below: they look for build/<ver>/shaders/jobs.json of the exe's own version first, then of the other one.

Shaders: .venv-py314\Scripts\python tools\shader_census.py --version <ver> --translate extracts every Maxwell shader to build\<ver>\shaders\ and runs it through the translator and spirv-val; failure groups go to failures.txt. tools\xlate_smoke.py checks the translator alone, without game files.

Reverse engineering (optional)

  • Ghidra 12.1.2 with Adubbz's Ghidra-Switch-Loader in toolchain\ghidra\ (JDK 21) opens the game modules (NSO).
  • tools\fetch_refs.sh clones the upstream projects this port builds on into refs\, read-only.