bootstrap is broken in rust-1.75.0-beta with: thread 'main' panicked at src/core/builder.rs:1795:26: std::fs::read_dir(registry_src) failed with No such file or directory (os error 2)

[stefson]

this is my attempt to bootstrap rust-1.75.0 beta, which fails with this beauty:

running sanity check
learning about cargo
> Assemble { target_compiler: Compiler { stage: 1, host: x86_64-gentoo-linux-musl } }
  > Assemble { target_compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl } }
  < Assemble { target_compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl } }
  > Rustc { target: x86_64-gentoo-linux-musl, compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, crates: [] }
    > Std { target: x86_64-gentoo-linux-musl, compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, crates: [], force_recompile: false, extra_rust_args: [] }
      > StartupObjects { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, target: x86_64-gentoo-linux-musl }
      < StartupObjects { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, target: x86_64-gentoo-linux-musl }
      c Assemble { target_compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl } }
      > Libdir { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, target: x86_64-gentoo-linux-musl }
        > Sysroot { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, force_recompile: false }
Removing sysroot /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/build/x86_64-gentoo-linux-musl/stage0-sysroot to avoid caching bugs
        < Sysroot { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, force_recompile: false }
Removing sysroot /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/build/x86_64-gentoo-linux-musl/stage0-sysroot/lib/rustlib/x86_64-gentoo-linux-musl/lib to avoid caching bugs
        c Sysroot { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, force_recompile: false }
      < Libdir { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, target: x86_64-gentoo-linux-musl }
      c Libdir { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, target: x86_64-gentoo-linux-musl }
      c Sysroot { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, force_recompile: false }
thread 'main' panicked at src/core/builder.rs:1795:26:
std::fs::read_dir(registry_src) failed with No such file or directory (os error 2)
stack backtrace:
   0:     0x564e4c278d2c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h05bd3f3cfb4e4d10
   1:     0x564e4c2a732c - core::fmt::write::h48e8631484ee755d
   2:     0x564e4c27402e - std::io::Write::write_fmt::h0a6193b9593cb997
   3:     0x564e4c278b14 - std::sys_common::backtrace::print::he91a2fc2cccbc13b
   4:     0x564e4c27ac93 - std::panicking::default_hook::{{closure}}::h7e7cca84ab3b6a5a
   5:     0x564e4c27a9b4 - std::panicking::default_hook::hb9acc10797035fd2
   6:     0x564e4c27b215 - std::panicking::rust_panic_with_hook::habf92a251463da55
   7:     0x564e4c27b111 - std::panicking::begin_panic_handler::{{closure}}::ha11af79f81437361
   8:     0x564e4c279256 - std::sys_common::backtrace::__rust_end_short_backtrace::hbbd2d14762e87dfc
   9:     0x564e4c27ae62 - rust_begin_unwind
  10:     0x564e4b71d8f5 - core::panicking::panic_fmt::he0005c8db2b53995
  11:     0x564e4b95a0d9 - bootstrap::core::builder::Builder::cargo::h81a7b3de28be0597
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:1795:26
  12:     0x564e4b784d13 - <bootstrap::core::build_steps::compile::Std as bootstrap::core::builder::Step>::run::hef99c80683adc61d
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/build_steps/compile.rs:209:25
  13:     0x564e4bc86e44 - bootstrap::core::builder::Builder::ensure::haf82eda529e87164
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:2142:23
  14:     0x564e4b78c76c - <bootstrap::core::build_steps::compile::Rustc as bootstrap::core::builder::Step>::run::h64561f053b21cfc9
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/build_steps/compile.rs:778:9
  15:     0x564e4bbf0e54 - bootstrap::core::builder::Builder::ensure::h58213f8b5515ed44
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:2142:23
  16:     0x564e4b794564 - <bootstrap::core::build_steps::compile::Assemble as bootstrap::core::builder::Step>::run::h8d0bf6ea1c5a27d1
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/build_steps/compile.rs:1594:9
  17:     0x564e4bc4ee83 - bootstrap::core::builder::Builder::ensure::h8cec5a594764982d
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:2142:23
  18:     0x564e4b950291 - bootstrap::core::builder::Builder::compiler::hfa7dd6787e9d598f
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:974:9
  19:     0x564e4b7843f4 - <bootstrap::core::build_steps::compile::Std as bootstrap::core::builder::Step>::make_run::h6d87d340d03044f2
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/build_steps/compile.rs:107:23
  20:     0x564e4b94a258 - bootstrap::core::builder::StepDescription::maybe_run::hb650c23b3ff31307
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:316:13
  21:     0x564e4b94afce - bootstrap::core::builder::StepDescription::run::hd04123386ea2e415
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:355:21
  22:     0x564e4b950223 - bootstrap::core::builder::Builder::run_step_descriptions::h6f290e94efac2e8f
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:966:9
  23:     0x564e4b94ffc6 - bootstrap::core::builder::Builder::execute_cli::hc0858ad5f355b7ee
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:947:9
  24:     0x564e4b9fb4cd - bootstrap::Build::build::h3bab7e9d07c88e90
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/lib.rs:670:17
  25:     0x564e4b720dcd - bootstrap::main::h1f0cea8e1c30d35c
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/bin/main.rs:74:5
  26:     0x564e4b71eb83 - core::ops::function::FnOnce::call_once::h43f31234d165b8e7
                               at /rustc/22d4a4ff135fd79c06cf0274754bd571b1fff400/library/core/src/ops/function.rs:250:5
  27:     0x564e4b71e406 - std::sys_common::backtrace::__rust_begin_short_backtrace::h59eef5b99cf3c0cb
                               at /rustc/22d4a4ff135fd79c06cf0274754bd571b1fff400/library/std/src/sys_common/backtrace.rs:154:18
  28:     0x564e4b71e4f9 - std::rt::lang_start::{{closure}}::had666b1fe66fcdca
                               at /rustc/22d4a4ff135fd79c06cf0274754bd571b1fff400/library/std/src/rt.rs:166:18
  29:     0x564e4c26b0d2 - std::rt::lang_start_internal::h0da414b7d71fa485
  30:     0x564e4b71e4d7 - std::rt::lang_start::hf899673c0c6d1ce0
                               at /rustc/22d4a4ff135fd79c06cf0274754bd571b1fff400/library/std/src/rt.rs:165:17
  31:     0x564e4b721c3e - main
  32:     0x7f18b57f2f1b - <unknown>
Traceback (most recent call last):
  File "/var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/./x.py", line 50, in <module>
    bootstrap.main()
  File "/var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 1134, in main
    bootstrap(args)
  File "/var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 1110, in bootstrap
    run(args, env=env, verbose=build.verbose, is_bootstrap=True)
  File "/var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 187, in run
    raise RuntimeError(err)

this is my command:

python3.11 ./x.py build -vv --config="/var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc_beta_src//config.toml -j12 || die

I can workaround this by reverting #115872

@pietroalbini can you please have a look at this error and the backtrace? I don't get why this error comes up. Thanks

[stefson]

this is my config.toml:

[llvm]
download-ci-llvm = false
optimize = true
release-debuginfo = false
assertions = false
ninja = true
targets = "X86"
experimental-targets = ""
link-shared = true
[build]
build = "x86_64-gentoo-linux-musl"
host = ["x86_64-gentoo-linux-musl"]
target = ["x86_64-gentoo-linux-musl"]
cargo = "/usr/lib/rust/1.74.0/bin/cargo"
rustc = "/usr/lib/rust/1.74.0/bin/rustc"
docs = false
compiler-docs = false
submodules = false
python = "python3.11"
locked-deps = true
vendor = true
extended = true
tools = ["rustfmt","cargo","rustdoc"]
verbose = 2
sanitizers = false
profiler = false
cargo-native-static = false
[install]
prefix = "/usr/lib/rust/1.75.0_beta"
sysconfdir = "etc"
docdir = "share/doc/rust"
bindir = "bin"
libdir = "lib"
mandir = "share/man"
[rust]
# https://github.com/rust-lang/rust/issues/54872
codegen-units-std = 1
optimize = true
debug = false
debug-assertions = false
debuginfo-level-rustc = 0
backtrace = true
incremental = false
default-linker = "x86_64-gentoo-linux-musl-gcc"
parallel-compiler = false
channel = "stable"
rpath = false
verbose-tests = true
optimize-tests = true
codegen-tests = true
dist-src = false
remap-debuginfo = true
lld = false
backtrace-on-ice = true
jemalloc = false
lto = "off"
[dist]
src-tarball = false
compression-formats = ["xz"]
[target.x86_64-gentoo-linux-musl]
cc = "x86_64-gentoo-linux-musl-gcc"
cxx = "x86_64-gentoo-linux-musl-g++"
linker = "x86_64-gentoo-linux-musl-gcc"
ar = "x86_64-gentoo-linux-musl-ar"
crt-static = false
llvm-config = "/usr/lib/llvm/17/bin/llvm-config"

[bjorn3]

The fixme here seems to be the issue:

rust/src/bootstrap/src/core/builder.rs

Lines 1792 to 1795 in 5526682

	// FIXME: handle vendored sources
	let registry_src = t!(home::cargo_home()).join("registry").join("src");
	let mut env_var = OsString::new();
	for entry in t!(std::fs::read_dir(registry_src)) {

As workaround you can try to run mkdir -p ~/.cargo/registry/src before the build.

cc @pietroalbini

[stefson]

I'm compiling offline, from the beta src tarball.

[bjorn3]

I'm aware. It seems that there is a bug when using vendored sources in combination with remap-debuginfo. Creating the ~/.cargo/registry/src directory should help even if it is kept empty.

[apiraino]

WG-prioritization assigning priority (Zulip discussion).

@rustbot label -I-prioritize +P-medium

[stefson]

Well, the cargo eclass is already exporting CARGO_HOME env var, and a CARGO_REGISTRY_DIR as well, and I'm doubtfull about the package manager allowing x.py to escape it's sandbox. Neverthelss, I'll try your suggestion when I'm back home on the machine.

[bjorn3]

If you have CARGO_HOME set the directory you need to create is $CARGO_HOME/.cargo/registry/src.