Can't build on Windows 10 with `x86_64-pc-windows-gnu`; GHA not actually cross-compiling either?

[EndilWayfare]

Following the instructions in the README, everything seemed to be working fine until ./y.rs build.

The MSVC toolchain is not yet supported by rustc_codegen_cranelift.
Switch to the MinGW toolchain for Windows support.
Hint: You can use `rustup set default-host x86_64-pc-windows-gnu` to
set the global default target to MinGW

Ok, fair enough. Compatibility layer time!

error: linking with `x86_64-w64-mingw32-gcc` failed: exit code: 1
  |
  = note: "x86_64-w64-mingw32-gcc" "-Wl,C:\\Users\\{redacted}\\AppData\\Local\\Temp\\rustczVob4a\\list.def" "-fno-use-linker-plugin" "-Wl,--dynamicbase" "-Wl,--disable-auto-image-base" "-m64" "-Wl,--high-entropy-va"
// lots and lots of paths
  = note: ld: cannot find -lcfgmgr32


warning: `rustc_codegen_cranelift` (lib) generated 1 warning
error: could not compile `rustc_codegen_cranelift` due to previous error; 1 warning emitted

Hmm, dang it. I can't seem to find helpful references to -lcfgmgr32 online. It sounds like mingw can't find the library it needs for cfgmgr32.dll. I tried reinstalling my mingw header package with pacman to no avail. I found out, as I understand it, that the x86_64-pc-windows-gnu toolchain uses a bundled mingw in a rust-mingw component, so my existing installation shouldn't matter anyway. What if I copy libcfgmgr32.a from my local mingw into the "lib" folder of the cargo source for the winapi it's referencing (that doesn't sound like the best idea...)? Huh, that... fixes it?

But now there's a new problem:

          C:\Users\{redacted}\rustc_codegen_cranelift\build_sysroot\target\x86_64-pc-windows-gnu\release\deps\std-f372f6b22d805bdf.std.53bbqtws-cgu.0.rcgu.o:std.53bbqtws-cgu.0:(.debug_info+0x40): relocation truncated to fit: IMAGE_REL_AMD64_ADDR32 against `.debug_str'

and other variations on that theme until

          C:\Users\{redacted}\rustc_codegen_cranelift\build_sysroot\target\x86_64-pc-windows-gnu\release\deps\std-f372f6b22d805bdf.std.53bbqtws-cgu.0.rcgu.o:std.53bbqtws-cgu.0:(.debug_info+0x56): additional relocation overflows omitted from the output
          collect2.exe: error: ld returned 1 exit status

I found this (https://github.com/bjorn3/rustc_codegen_cranelift/issues/977#issuecomment-830650448), which seems to be related, except I'm not using MSVC. (How do I disable DWARF debuginfo?) Based on that comment/thread, I tried to comment out the bit in y.rs that blows up on MSVC, just for laughs. No luck; not yet implemented: Unimplemented TLS model in x64 backend: Coff Yep, that's what the July report said, and it seems to still be the case. From what I can gather, the ADDR32 bit of it has something to do with jumps into relocatable code being too large to fit in a 32-bit pointer. I can confirm that my installed gnu toolchain compiles other crates successfully (I have nothing nearly as large as a sysroot, though).

I played around with a bunch of RUSTFLAGS combination (both to rustc and passthrough to gcc codegen/linking) based on tangential information (very little info on IMAGE_REL_AMD64_ADDR32 specifically) to no avail. It's possible that this is an effective direction, and I just don't know enough about gcc to make it work. It seems odd to me though, because there's no mention (that I could find) of special compiler args being necessary.

I tried a few scattershot previous commits, before and after the transition to a Rust-based build system. Same problem.

I tried cross-compiling with Ubuntu on WSL2. No luck:

❯ ./y.rs build
[BUILD] y.rs
info: syncing channel updates for 'nightly-2021-08-07-x86_64-pc-windows-gnu'
info: latest update on 2021-08-07, rust version 1.56.0-nightly (5ad7389bd 2021-08-06)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'llvm-tools-preview'
info: downloading component 'rust-docs'
info: downloading component 'rust-mingw'
info: downloading component 'rust-src'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: downloading component 'rustc-dev'
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'llvm-tools-preview'
info: installing component 'rust-docs'
info: installing component 'rust-mingw'
info: installing component 'rust-src'
info: installing component 'rust-std'
info: installing component 'rustc'
info: installing component 'rustc-dev'
info: installing component 'rustfmt'
error: the 'rustc' binary, normally provided by the 'rustc' component, is not applicable to the 'nightly-2021-08-07-x86_64-pc-windows-gnu' toolchain

What are you talking about? It says info: downloading component 'rustc' and then info: installing component 'rustc' RIGHT THERE. I have no idea what's going on, as I typically don't mess with nightly/betas.

I gave up on trying to compile from source and checked out the GHA artifacts. Looks like the uploading artifacts from the windows CI build is disabled? https://github.com/bjorn3/rustc_codegen_cranelift/blob/d498e6d6971575714353f11aa4d3e63a9d2030b2/.github/workflows/main.yml#L155-L162

Oh well, cross-x86_64-mingw sounds promising.

$ ./cargo
bash: ./cargo: cannot execute binary file: Exec format error

Dang it.

$ file cargo 
cargo: ELF 64-bit LSB shared object x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=b8e72eb3363cc311d848656c88fa225cbaf411a7, for GNU/Linux 3.2.0, with debug_info, not stripped

Well, that definitely doesn't sound right.

./y.rs build --sysroot none did compile, although it (understandably) does nothing and hangs forever when invoked. However:

$ file build/cargo
build/cargo: PE32+ executable (console) x86-64, for MS Windows

Unless I'm doing something wrong (quite possible), it seems that the cross-compile CI target isn't measuring what it's supposed to be measuring (successfully producing a Windows-runnable artifact from Linux).

TWIR sent me here. I've been following this on-and-off for a while, and it's really awesome to see how close it is to basically-feature-complete. I wanted to get in on it (faster debug builds? Yes please!), but I've tried as hard as I can on my own.

[EndilWayfare]

Ayyyy, got it to work.

TL;DR: Compiling on Windows == still broken for me. Cross-compiling from Linux works if 1) target/host overrides are set correctly and 2) you make sure target_triple is propagated to compiling the cargo wrapper.

Cross-compiling from WSL wasn't working because somehow in all my faffing about I overrode the default host triple in addition to the default target. Oof. So, it compiled then, but the resulting cargo failed in Windows with the same error (and file output) as the GHA artifact did. After digging around in the source code a bit, since the problem seemed to hinge on the cargo binary, I decided to try altering this:
https://github.com/bjorn3/rustc_codegen_cranelift/blob/d498e6d6971575714353f11aa4d3e63a9d2030b2/build_system/build_sysroot.rs#L46-L53
to this:

    // Build and copy cargo wrapper
    let mut build_cargo_wrapper_cmd = Command::new("rustc");
    build_cargo_wrapper_cmd
        .arg("--target")
        .arg(target_triple)
        .arg("scripts/cargo.rs")
        .arg("-o")
        .arg(target_dir.join("cargo"))
        .arg("-g");
    spawn_and_wait(build_cargo_wrapper_cmd);

, to match https://github.com/bjorn3/rustc_codegen_cranelift/blob/d498e6d6971575714353f11aa4d3e63a9d2030b2/build_system/build_sysroot.rs#L179-L181

That would also explain why CI doesn't appear to be producing a viable cross-compiled artifact!

[EndilWayfare]

Dammit, just kidding. That was my system cargo.exe, not ./build/cargo.exe. The new cross-compiled cargo hangs forever like the ./y.rs build --sysroot none version...

[bjorn3]

Cross-compiling from WSL wasn't working because somehow in all my faffing about I overrode the default host triple in addition to the default target. Oof. So, it compiled then, but the resulting cargo failed in Windows with the same error (and file output) as the GHA artifact did. After digging around in the source code a bit, since the problem seemed to hinge on the cargo binary, I decided to try altering this:

I have never actually tried to cross-compile cg_clif itself for windows. CI tests compiling cg_clif on windows using the mingw toolchain and it tests compiling programs using cg_clif for windows using the mingw toolchain.

But now there's a new problem:

          C:\Users\{redacted}\rustc_codegen_cranelift\build_sysroot\target\x86_64-pc-windows-gnu\release\deps\std-f372f6b22d805bdf.std.53bbqtws-cgu.0.rcgu.o:std.53bbqtws-cgu.0:(.debug_info+0x40): relocation truncated to fit: IMAGE_REL_AMD64_ADDR32 against `.debug_str'

and other variations on that theme until

          C:\Users\{redacted}\rustc_codegen_cranelift\build_sysroot\target\x86_64-pc-windows-gnu\release\deps\std-f372f6b22d805bdf.std.53bbqtws-cgu.0.rcgu.o:std.53bbqtws-cgu.0:(.debug_info+0x56): additional relocation overflows omitted from the output
          collect2.exe: error: ld returned 1 exit status

Maybe removing https://github.com/bjorn3/rustc_codegen_cranelift/blob/d498e6d6971575714353f11aa4d3e63a9d2030b2/build_sysroot/Cargo.toml#L23 helps?

[EndilWayfare]

I have never actually tried to cross-compile cg_clif itself for windows. CI tests compiling cg_clif on windows using the mingw toolchain and it tests compiling programs using cg_clif for windows using the mingw toolchain.

Yeah, I figured that out yesterday, I just ran out of time trying weird combinations of compiling different bits on Windows and WSL to post about it. I was like, OOOOOH got it, the host/target are operating on a different layer of indirection than I assumed. It's not that the codegen is cross-compiled to run on windows, it's that the codegen does cross-compilation (running on Linux, producing cranelift debug builds that run on windows). That's awesome, but I don't feel like doubling up on dependency sources/build-artifacts between WSL and Windows...

[EndilWayfare]

Maybe removing

https://github.com/bjorn3/rustc_codegen_cranelift/blob/d498e6d6971575714353f11aa4d3e63a9d2030b2/build_sysroot/Cargo.toml#L23

helps?

Ahhhh, the Cargo.toml in "build_sysroot", ok. I messed around with the root one, but I missed that there was another. Well, it helps a little, in the sense that it actually compiles successfully. But, it also doesn't because the resulting "cargo.exe" hangs indefinitely still.

[EndilWayfare]

While I'm at it, is there a reason that the windows CI target is disabled from uploading its artifacts? Later today, I might get a chance to try compiling on a fresher, less battle-worn Windows dev environment and we'll see if that changes anything.

[bjorn3]

I can't remember. Maybe a leftover from trying to set up windows ci?

[EndilWayfare]

Ok, I decided to let cargo.exe sit there for a while and find out if it ever actually does anything. It does. It allocates a lot of memory until it gets OOM'd.

$ build/cargo.exe
memory allocation of 1766464 bytes failed
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 1455, kind: Uncategorized, message: "The paging file is too small for this operation to complete." }', scripts/cargo.rs:76:50

Huh. Ok, once more with RUST_BACKTRACE=1. While we're waiting for it to explode, let's pull up Task Manager and see if that tells us anything.

Hmmm, cargo.exe is... multiplying? Those are all subprocesses under the terminal emulator that initially invoked cargo.exe...

!?

It caps out at over 9,000 [yes, really] subprocesses consuming 19 GB of RAM before it goes down.
Interestingly, it doesn't actually give me a backtrace. It just goes

$ build/cargo.exe
memory allocation of 2269140 bytes failed

It seems to slowly unwind the subprocesses after it prints the failure message, and it doesn't "return" to the shell prompt until all the subprocesses are gone.

[EndilWayfare]

Interestingly, it doesn't actually give me a backtrace

oooooh, could that be a side-effect of debug = true?

[EndilWayfare]

It's also very probable that the windows CI build suffers from the same issue. The action only tests whether it compiles; test.sh isn't run, because windows.

I tried running test.sh locally with my MinGW bash, and it exploded. First, I had to add "./build/bin" and "/c/Users/{user}/.rustup/toolchains/nightly-2021-08-07-x86_64-pc-windows-gnu/bin" to $PATH, because otherwise windows can't find the .dll's. Then, [BUILD] mini_core failed with the same relocation truncated to fit: IMAGE_REL_AMD64_ADDR32 nonsense, so I added -C debuginfo=0 to the line in "scripts/tests.sh". Now mini_core gets into rustc proper, but it ICEs on an unresolved symbol.

thread 'rustc' panicked at 'can't resolve symbol _ZN9mini_core8A_STATIC17h1e90a401893c503aE', C:\Users\pormand\.cargo\git\checkouts\wasmtime-41807828cb3a7a7e\9c550fc\cranelift\jit\src\backend.rs:288:21
stack backtrace:
   0:         0x6f2d2d2f - std::backtrace_rs::backtrace::dbghelp::trace::h15703146ef9749a7
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src\..\..\backtrace\src\backtrace/dbghelp.rs:98:5
   1:         0x6f2d2d2f - std::backtrace_rs::backtrace::trace_unsynchronized::hf8d56c17f1e136e9
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src\..\..\backtrace\src\backtrace/mod.rs:66:5
   2:         0x6f2d2d2f - std::sys_common::backtrace::_print_fmt::hab3c52876e76d449
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src\sys_common/backtrace.rs:67:5   
   3:         0x6f2d2d2f - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hab9c1cc6e7fc385b   
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src\sys_common/backtrace.rs:46:22  
   4:         0x6f33e39a - core::fmt::write::hae96daedf1d6be29
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\core\src\fmt/mod.rs:1117:17
   5:         0x6f2c6262 - std::io::Write::write_fmt::hafe5401ad6ee5626
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src\io/mod.rs:1667:15
   6:         0x6f2d64e0 - std::sys_common::backtrace::_print::hbe08c5b6a843015d
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src\sys_common/backtrace.rs:49:5   
   7:         0x6f2d64e0 - std::sys_common::backtrace::print::h674aa25aba0c71a3
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src\sys_common/backtrace.rs:36:9   
   8:         0x6f2d64e0 - std::panicking::default_hook::{{closure}}::h5c12d9d9abbca0f9
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src/panicking.rs:210:50
   9:         0x6f2d6094 - std::panicking::default_hook::hbc4ccd03703b88a3
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src/panicking.rs:227:9
  10:     0x7ff6672d28fe - cg_clif::DEFAULT_HOOK::{{closure}}::{{closure}}::h4825952fd8172fa7
  11:         0x6f2d6dc1 - std::panicking::rust_panic_with_hook::h1ecdc969fd899f2f
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src/panicking.rs:628:17
  12:         0x6f2d6899 - std::panicking::begin_panic_handler::{{closure}}::h36573f2e57c65019
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src/panicking.rs:521:13
  13:         0x6f2d373f - std::sys_common::backtrace::__rust_end_short_backtrace::hbff34e368d9c5643
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src\sys_common/backtrace.rs:141:18 
  14:         0x6f2d67f9 - rust_begin_unwind
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src/panicking.rs:517:5
  15:         0x6f2d67ac - std::panicking::begin_panic_fmt::h4b4cd9d4d947fbb4
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src/panicking.rs:460:5
  16:     0x7ffba855d144 - cranelift_jit::backend::JITModule::get_address::ha71330b8095d55ca
  17:     0x7ffba85639fa - cranelift_jit::compiled_blob::CompiledBlob::perform_relocations::h209dee35f48b61c5
  18:     0x7ffba855de36 - cranelift_jit::backend::JITModule::finalize_definitions::hca5b226777528b55
  19:     0x7ffba849e66f - rustc_codegen_cranelift::driver::jit::run_jit::h9a1e46bdfa469fae
  20:     0x7ffba8503f4d - <rustc_codegen_cranelift::CraneliftCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate::h495a51801e90fb82
  21:           0xadd650 - rustc_interface::passes::QueryContext::enter::hd0572fc530ac941d
  22:           0xacdc5b - rustc_interface::queries::Queries::ongoing_codegen::h2f921abc637d3d75
  23:           0x9df8ed - rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter::h1cb4948d439e9483        
  24:           0x99a85c - rustc_span::with_source_map::h6cc506dfc70b42ef
  25:           0x9e1120 - rustc_interface::interface::create_compiler_and_run::hf8038fe8f954301a
  26:           0x9a7b3d - std::sys_common::backtrace::__rust_begin_short_backtrace::hb1ec0507bfe8704c
  27:           0x99397e - core::ops::function::FnOnce::call_once{{vtable.shim}}::h5d71226d451acc81
  28:         0x6f2e4557 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h525f0061b701c388
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\library\alloc\src/boxed.rs:1636:9
  29:         0x6f2e4557 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h5e31eae84f71f0a7
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\library\alloc\src/boxed.rs:1636:9
  30:         0x6f2e4557 - std::sys::windows::thread::Thread::new::thread_start::h1ed0a385a606a823
                               at /rustc/5ad7389bdd1abe7d2c6f73a233af1a7a69e96285\/library\std\src\sys\windows/thread.rs:57:17
  31:     0x7ffc41e57034 - <unknown>
  32:     0x7ffc43782651 - <unknown>

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/bjorn3/rustc_codegen_cranelift/issues/new

note: rustc 1.56.0-nightly (5ad7389bd 2021-08-06) running on x86_64-pc-windows-gnu

note: compiler flags: -Z unstable-options -C debuginfo=2 -C llvm-args=mode=jit -C prefer-dynamic

query stack during panic:
end of query stack

Sounds suspiciously like https://github.com/bjorn3/rustc_codegen_cranelift/issues/1137?

[bjorn3]

Hmmm, cargo.exe is... multiplying? Those are all subprocesses under the terminal emulator that initially invoked cargo.exe...

I see what is going on. On Unix systems the directory containing the current executabe is not in PATH, so trying to spawn "cargo" will attempt to run the original cargo. On windows it is in PATH, so it will attempt to run the wrapper itself.

https://github.com/bjorn3/rustc_codegen_cranelift/blob/d498e6d6971575714353f11aa4d3e63a9d2030b2/scripts/cargo.rs#L76

[bjorn3]

I tried running test.sh locally with my MinGW bash, and it exploded.

Looks like the jit tests don't work. Could you try if disabling those tests will make the rest of the tests pass?

Then, [BUILD] mini_core failed with the same relocation truncated to fit: IMAGE_REL_AMD64_ADDR32 nonsense, so I added -C debuginfo=0 to the line in "scripts/tests.sh".

Todo:

• Disable dwarf debuginfo generation on Windows.

[EndilWayfare]

I see what is going on. On Unix systems the directory containing the current executabe is not in PATH, so trying to spawn "cargo" will attempt to run the original cargo. On windows it is in PATH, so it will attempt to run the wrapper itself.

Confirmed. Renaming from cargo.exe to cargo-clif.exe works. It successfully substitutes for cargo to build another crate, except I have to set RUSTFLAGS="-C debuginfo=0" or IMAGE_REL_AMD64_ADDR32 happens.

[EndilWayfare]

Looks like the jit tests don't work. Could you try if disabling those tests will make the rest of the tests pass?

It looks like disabling jit tests and tests that invoke cargo works, and the other tests succeed. I say "looks", because I haven't gotten to the end of "tests.sh" with modification yet. It's a lot harder to kill the runaway-process phenomenon when it's a binary invoking cargo rather than bash invoking cargo; it's enough to SIGINT in bash, but you kind of have to wait for OOM in the other case because Task Manager gets overwhelmed and unusable.

[EndilWayfare]

base_sysroot_tests completes successfully, under above caveats