ICE: `git pull upstream master` + `./x build` fails

[whtahy]

Have been running into ICE during ./x build run right after git pull upstream master inside my fork of the rust repo:

• ICE when fork is maybe 50+ commits behind (ie, 1 day)
• No ICE when fork is small number of commits behind, maybe 5-10
• No ICE if I run ./x clean before ./x build

My config.toml is set to

profile = "compiler"
changelog-seen = 2

Code

git pull upstream master
./x build

Meta

rustc --version --verbose inside the rust\build\bootstrap\debug folder:

rustc 1.68.2 (9eb3afe9e 2023-03-27)
binary: rustc
commit-hash: 9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0
commit-date: 2023-03-27
host: x86_64-pc-windows-msvc
release: 1.68.2
LLVM version: 15.0.6

Error output

Compiling paths v0.0.0 (Z:\git\rust\src\tools\rust-analyzer\crates\paths)
thread 'rustc' panicked at 'assertion failed: value <= 0x7FFF_FFFF', Z:\git\rust\compiler\rustc_query_system\src\dep_graph\serialized.rs:29:1
stack backtrace:
thread 'rustc' panicked at 'assertion failed: value <= 0x7FFF_FFFF', Z:\git\rust\compiler\rustc_query_system\src\dep_graph\serialized.rs:29:1
stack backtrace:
thread 'rustc' panicked at 'index out of bounds: the len is 409101 but the index is 409101', Z:\git\rust\compiler\rustc_serialize\src\leb128.rs:82:1
stack backtrace:
   Compiling proc-macro-srv v0.0.0 (Z:\git\rust\src\tools\rust-analyzer\crates\proc-macro-srv)
   Compiling rustc-ap-rustc_lexer v727.0.0
thread 'rustc' panicked at 'assertion failed: value <= 0x7FFF_FFFF', Z:\git\rust\compiler\rustc_query_system\src\dep_graph\serialized.rs:29:1

Backtrace

stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic
   3: RNvXs1n_NtCscw00T8nT1nk_15rustc_serialize9serializeTNtNtNtCsjVv1O0u6BHl_18rustc_query_system9dep_graph10serialized22SerializedDepNodeIndexNtNtCs6USR5qbnuSD_16rustc_query_impl13on_disk_cache15AbsoluteBytePosEINtB6_9DecodableNtNtB8_6opaque10MemDecoderE6deco
   4: RNvXsf_NtCscw00T8nT1nk_15rustc_serialize9serializeINtNtCs7xoPx3WEF6k_5alloc3vec3VecTNtNtNtCsjVv1O0u6BHl_18rustc_query_system9dep_graph10serialized22SerializedDepNodeIndexNtNtCs6USR5qbnuSD_16rustc_query_impl13on_disk_cache15AbsoluteBytePosEEINtB5_9Decodabl
   5: rustc_query_impl::on_disk_cache::decode_tagged::<rustc_serialize::opaque::MemDecoder, u128, rustc_query_impl::on_disk_cache::Footer>
   6: <rustc_query_impl::on_disk_cache::OnDiskCache as rustc_middle::ty::context::OnDiskCache>::new
   7: rustc_incremental::persist::load::load_query_result_cache::<rustc_query_impl::on_disk_cache::OnDiskCache>
   8: rustc_interface::passes::create_global_ctxt
   9: <rustc_interface::queries::Query<&rustc_middle::ty::context::GlobalCtxt>>::compute::<<rustc_interface::queries::Queries>::global_ctxt::{closure#0}>
  10: RINvMs5_NtCshAsjlLtZwvs_15rustc_interface7queriesNtNtB8_9interface8Compiler5enterNCNCNvCsfeZZ0NiaWIf_17rustc_driver_impl12run_compilers_0s0_0INtNtCs5JpM6iT4JTm_4core6result6ResultINtNtB2k_6option6OptionNtB6_6LinkerENtCsa3NkH8BRZw7_10rustc_span15ErrorGuara
  11: rustc_span::set_source_map::<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}::{closure#0}>
  12: RINvMs_CsiQv3wPN5z1p_10scoped_tlsINtB5_9ScopedKeyNtCsa3NkH8BRZw7_10rustc_span14SessionGlobalsE3setNCINvNtCshAsjlLtZwvs_15rustc_interface9interface12run_compilerINtNtCs5JpM6iT4JTm_4core6result6ResultuNtBN_15ErrorGuaranteedENCNvCsfeZZ0NiaWIf_17rustc_driver_
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.71.0-dev running on x86_64-pc-windows-msvc

note: compiler flags: --crate-type lib -C opt-level=3 -C embed-bitcode=no -C debuginfo=0 -Z unstable-options -C incremental=[REDACTED] -C symbol-mangling-version=v0 -Z unstable-options -Z macro-backtrace -C split-debuginfo=packed -C target-feature=+crt-static -C llvm-args=-import-instr-limit=10 -Z binary-dep-depinfo -Z tls-model=initial-exec -Z allow-features=binary-dep-depinfo,proc_macro_span,proc_macro_span_shrink,proc_macro_diagnostic,proc_macro_internals,proc_macro_diagnostic,proc_macro_span,proc_macro_span_shrink

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
   Compiling tracing-core v0.1.30
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic
   3: RNvXs1n_NtCscw00T8nT1nk_15rustc_serialize9serializeTNtNtNtCsjVv1O0u6BHl_18rustc_query_system9dep_graph10serialized22SerializedDepNodeIndexNtNtCs6USR5qbnuSD_16rustc_query_impl13on_disk_cache15AbsoluteBytePosEINtB6_9DecodableNtNtB8_6opaque10MemDecoderE6deco
   4: RNvXsf_NtCscw00T8nT1nk_15rustc_serialize9serializeINtNtCs7xoPx3WEF6k_5alloc3vec3VecTNtNtNtCsjVv1O0u6BHl_18rustc_query_system9dep_graph10serialized22SerializedDepNodeIndexNtNtCs6USR5qbnuSD_16rustc_query_impl13on_disk_cache15AbsoluteBytePosEEINtB5_9Decodabl
   5: rustc_query_impl::on_disk_cache::decode_tagged::<rustc_serialize::opaque::MemDecoder, u128, rustc_query_impl::on_disk_cache::Footer>
   6: <rustc_query_impl::on_disk_cache::OnDiskCache as rustc_middle::ty::context::OnDiskCache>::new
   Compiling itertools v0.10.5
   7: rustc_incremental::persist::load::load_query_result_cache::<rustc_query_impl::on_disk_cache::OnDiskCache>
   8: rustc_interface::passes::create_global_ctxt
   Compiling windows-sys v0.42.0
   9: <rustc_interface::queries::Query<&rustc_middle::ty::context::GlobalCtxt>>::compute::<<rustc_interface::queries::Queries>::global_ctxt::{closure#0}>
  10: RINvMs5_NtCshAsjlLtZwvs_15rustc_interface7queriesNtNtB8_9interface8Compiler5enterNCNCNvCsfeZZ0NiaWIf_17rustc_driver_impl12run_compilers_0s0_0INtNtCs5JpM6iT4JTm_4core6result6ResultINtNtB2k_6option6OptionNtB6_6LinkerENtCsa3NkH8BRZw7_10rustc_span15ErrorGuara
  11: rustc_span::set_source_map::<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}::{closure#0}>
  12: RINvMs_CsiQv3wPN5z1p_10scoped_tlsINtB5_9ScopedKeyNtCsa3NkH8BRZw7_10rustc_span14SessionGlobalsE3setNCINvNtCshAsjlLtZwvs_15rustc_interface9interface12run_compilerINtNtCs5JpM6iT4JTm_4core6result6ResultuNtBN_15ErrorGuaranteedENCNvCsfeZZ0NiaWIf_17rustc_driver_
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.71.0-dev running on x86_64-pc-windows-msvc

note: compiler flags: --crate-type lib -C opt-level=3 -C embed-bitcode=no -C debuginfo=0 -Z unstable-options -C incremental=[REDACTED] -C symbol-mangling-version=v0 -Z unstable-options -Z macro-backtrace -C split-debuginfo=packed -C target-feature=+crt-static -C llvm-args=-import-instr-limit=10 -Z binary-dep-depinfo -Z tls-model=initial-exec -Z allow-features=binary-dep-depinfo,proc_macro_span,proc_macro_span_shrink,proc_macro_diagnostic,proc_macro_internals,proc_macro_diagnostic,proc_macro_span,proc_macro_span_shrink

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
   Compiling lock_api v0.4.9
   Compiling memoffset v0.6.5
   Compiling indexmap v1.9.2
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic_bounds_check
   3: <u64 as rustc_serialize::serialize::Decodable<rustc_serialize::opaque::MemDecoder>>::decode
   4: RNvXs1n_NtCscw00T8nT1nk_15rustc_serialize9serializeTNtNtNtCsjVv1O0u6BHl_18rustc_query_system9dep_graph10serialized22SerializedDepNodeIndexNtNtCs6USR5qbnuSD_16rustc_query_impl13on_disk_cache15AbsoluteBytePosEINtB6_9DecodableNtNtB8_6opaque10MemDecoderE6deco
   5: RNvXsf_NtCscw00T8nT1nk_15rustc_serialize9serializeINtNtCs7xoPx3WEF6k_5alloc3vec3VecTNtNtNtCsjVv1O0u6BHl_18rustc_query_system9dep_graph10serialized22SerializedDepNodeIndexNtNtCs6USR5qbnuSD_16rustc_query_impl13on_disk_cache15AbsoluteBytePosEEINtB5_9Decodabl
   6: rustc_query_impl::on_disk_cache::decode_tagged::<rustc_serialize::opaque::MemDecoder, u128, rustc_query_impl::on_disk_cache::Footer>
   7: <rustc_query_impl::on_disk_cache::OnDiskCache as rustc_middle::ty::context::OnDiskCache>::new
   8: rustc_incremental::persist::load::load_query_result_cache::<rustc_query_impl::on_disk_cache::OnDiskCache>
   9: rustc_interface::passes::create_global_ctxt
  10: <rustc_interface::queries::Query<&rustc_middle::ty::context::GlobalCtxt>>::compute::<<rustc_interface::queries::Queries>::global_ctxt::{closure#0}>
  11: RINvMs5_NtCshAsjlLtZwvs_15rustc_interface7queriesNtNtB8_9interface8Compiler5enterNCNCNvCsfeZZ0NiaWIf_17rustc_driver_impl12run_compilers_0s0_0INtNtCs5JpM6iT4JTm_4core6result6ResultINtNtB2k_6option6OptionNtB6_6LinkerENtCsa3NkH8BRZw7_10rustc_span15ErrorGuara
  12: rustc_span::set_source_map::<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}::{closure#0}>
  13: RINvMs_CsiQv3wPN5z1p_10scoped_tlsINtB5_9ScopedKeyNtCsa3NkH8BRZw7_10rustc_span14SessionGlobalsE3setNCINvNtCshAsjlLtZwvs_15rustc_interface9interface12run_compilerINtNtCs5JpM6iT4JTm_4core6result6ResultuNtBN_15ErrorGuaranteedENCNvCsfeZZ0NiaWIf_17rustc_driver_
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.71.0-dev running on x86_64-pc-windows-msvc

note: compiler flags: --crate-type lib -C opt-level=3 -C embed-bitcode=no -C debuginfo=0 -Z unstable-options -C incremental=[REDACTED] -C symbol-mangling-version=v0 -Z unstable-options -Z macro-backtrace -C split-debuginfo=packed -C target-feature=+crt-static -C llvm-args=-import-instr-limit=10 -Z binary-dep-depinfo -Z tls-model=initial-exec -Z allow-features=binary-dep-depinfo,proc_macro_span,proc_macro_span_shrink,proc_macro_diagnostic,proc_macro_internals,proc_macro_diagnostic,proc_macro_span,proc_macro_span_shrink

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
   Compiling always-assert v0.1.2
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic
   3: RNvXs1n_NtCscw00T8nT1nk_15rustc_serialize9serializeTNtNtNtCsjVv1O0u6BHl_18rustc_query_system9dep_graph10serialized22SerializedDepNodeIndexNtNtCs6USR5qbnuSD_16rustc_query_impl13on_disk_cache15AbsoluteBytePosEINtB6_9DecodableNtNtB8_6opaque10MemDecoderE6deco
   4: RNvXsf_NtCscw00T8nT1nk_15rustc_serialize9serializeINtNtCs7xoPx3WEF6k_5alloc3vec3VecTNtNtNtCsjVv1O0u6BHl_18rustc_query_system9dep_graph10serialized22SerializedDepNodeIndexNtNtCs6USR5qbnuSD_16rustc_query_impl13on_disk_cache15AbsoluteBytePosEEINtB5_9Decodabl
   5: rustc_query_impl::on_disk_cache::decode_tagged::<rustc_serialize::opaque::MemDecoder, u128, rustc_query_impl::on_disk_cache::Footer>
   6: <rustc_query_impl::on_disk_cache::OnDiskCache as rustc_middle::ty::context::OnDiskCache>::new
   7: rustc_incremental::persist::load::load_query_result_cache::<rustc_query_impl::on_disk_cache::OnDiskCache>
   8: rustc_interface::passes::create_global_ctxt
   9: <rustc_interface::queries::Query<&rustc_middle::ty::context::GlobalCtxt>>::compute::<<rustc_interface::queries::Queries>::global_ctxt::{closure#0}>
  10: RINvMs5_NtCshAsjlLtZwvs_15rustc_interface7queriesNtNtB8_9interface8Compiler5enterNCNCNvCsfeZZ0NiaWIf_17rustc_driver_impl12run_compilers_0s0_0INtNtCs5JpM6iT4JTm_4core6result6ResultINtNtB2k_6option6OptionNtB6_6LinkerENtCsa3NkH8BRZw7_10rustc_span15ErrorGuara
  11: rustc_span::set_source_map::<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}::{closure#0}>
  12: RINvMs_CsiQv3wPN5z1p_10scoped_tlsINtB5_9ScopedKeyNtCsa3NkH8BRZw7_10rustc_span14SessionGlobalsE3setNCINvNtCshAsjlLtZwvs_15rustc_interface9interface12run_compilerINtNtCs5JpM6iT4JTm_4core6result6ResultuNtBN_15ErrorGuaranteedENCNvCsfeZZ0NiaWIf_17rustc_driver_
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.71.0-dev running on x86_64-pc-windows-msvc

note: compiler flags: --crate-type bin -C embed-bitcode=no -C debuginfo=0 -C debug-assertions=off -Z unstable-options -C incremental=[REDACTED] -Z binary-dep-depinfo -C target-feature=+crt-static -Z unstable-options -Z allow-features=binary-dep-depinfo,proc_macro_span,proc_macro_span_shrink,proc_macro_diagnostic,proc_macro_internals,proc_macro_diagnostic,proc_macro_span,proc_macro_span_shrink

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
error: could not compile `limit`

Caused by:
  process didn't exit successfully: `Z:\git\rust\build\bootstrap\debug\rustc --crate-name limit --edition=2021 'crates\limit\src\lib.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=0 --cfg 'feature="default"' --cfg 'feature="tracking"' -Zunstable-options --check-cfg 'values(feature, "default", "tracking")' --check-cfg 'names()' --check-cfg 'values()' -C metadata=02424dccf3e833fa -C extra-filename=-02424dccf3e833fa --out-dir 'Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -C 'incremental=Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\incremental' -L 'dependency=Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps' -L 'dependency=Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\release\deps' -Csymbol-mangling-version=v0 -Zunstable-options '--check-cfg=values(bootstrap)' -Zmacro-backtrace -Csplit-debuginfo=packed -Ctarget-feature=+crt-static -Cllvm-args=-import-instr-limit=10 -Z binary-dep-depinfo` (exit code: 0x80000003)
warning: build failed, waiting for other jobs to finish...
error: could not compile `la-arena`

Caused by:
  process didn't exit successfully: `Z:\git\rust\build\bootstrap\debug\rustc --crate-name la_arena --edition=2021 'lib\la-arena\src\lib.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=0 -Zunstable-options --check-cfg 'values(feature)' --check-cfg 'names()' --check-cfg 'values()' -C metadata=260de9a8730fc430 -C extra-filename=-260de9a8730fc430 --out-dir 'Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -C 'incremental=Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\incremental' -L 'dependency=Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps' -L 'dependency=Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\release\deps' -Csymbol-mangling-version=v0 -Zunstable-options '--check-cfg=values(bootstrap)' -Zmacro-backtrace -Csplit-debuginfo=packed -Ctarget-feature=+crt-static -Cllvm-args=-import-instr-limit=10 -Z binary-dep-depinfo` (exit code: 0x80000003)
error: could not compile `paths`

Caused by:
  process didn't exit successfully: `Z:\git\rust\build\bootstrap\debug\rustc --crate-name paths --edition=2021 'crates\paths\src\lib.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=0 -Zunstable-options --check-cfg 'values(feature)' --check-cfg 'names()' --check-cfg 'values()' -C metadata=8e46147320c94967 -C extra-filename=-8e46147320c94967 --out-dir 'Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps' --target x86_64-pc-windows-msvc -C 'incremental=Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\incremental' -L 'dependency=Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps' -L 'dependency=Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\release\deps' -Csymbol-mangling-version=v0 -Zunstable-options '--check-cfg=values(bootstrap)' -Zmacro-backtrace -Csplit-debuginfo=packed -Ctarget-feature=+crt-static -Cllvm-args=-import-instr-limit=10 -Z binary-dep-depinfo` (exit code: 0x80000003)
error: could not compile `proc-macro-srv`

Caused by:
  process didn't exit successfully: `Z:\git\rust\build\bootstrap\debug\rustc --crate-name build_script_build --edition=2021 'crates\proc-macro-srv\build.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=0 -C debug-assertions=off --cfg 'feature="sysroot-abi"' -Zunstable-options --check-cfg 'values(feature, "sysroot-abi")' --check-cfg 'names()' --check-cfg 'values()' -C metadata=d6868d2a848856da -C extra-filename=-d6868d2a848856da --out-dir 'Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\release\build\proc-macro-srv-d6868d2a848856da' -C 'incremental=Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\release\incremental' -L 'dependency=Z:\git\rust\build\x86_64-pc-windows-msvc\stage1-tools\release\deps' -Z binary-dep-depinfo` (exit code: 0x80000003)

[whtahy]

Looks similar/same as issue #108745?

[saethlin]

I think this is the general problem described in #76720

I recently authored a number of changes to the on-disk format that incremental builds use, such as #110083 and I saw every kind of ICE like this when switching to or from that branch.

[jyn514]

Yes, the backtrace shows this is a stage 1 build, not stage 0 (the version is 1.71-dev).

[whtahy]

I think this is the general problem described in #76720

I recently authored a number of changes to the on-disk format that incremental builds use, such as #110083 and I saw every kind of ICE like this when switching to or from that branch.

Aha, thanks for the pointer!

Yes, the backtrace shows this is a stage 1 build, not stage 0 (the version is 1.71-dev).

Thanks, will follow up in the other issue.