Skip to content

Use of tempfile for lib.def leads to non-reproducible PDBs #71714

@nikhilm

Description

@nikhilm

I've been working with a rust code base that we are trying to use a shared cache for (Bazel FWIW). As part of that I've been looking at truly bit-identical build outputs for a given set of inputs.

There are plenty of problems with PDB files on Windows (which contain debug info) because they tend to encode absolute paths to the object files and other resources they were created from. I've managed to eliminate most of these by using deterministic build directories across all machines, and using tempdir's that are a hash of something unique. There is just one more bit that I can't control directly, which is this use of tmpdir.join("lib.def")

let path = tmpdir.join("lib.def");
. This ends up putting a few random characters in the path that throws off the PDB.

I'm not entirely sure what a good solution will look like. I'm not familiar enough with compiler internals to know if some deterministic location based on some unique values can be used, or alternatively if this can be exposed as a compiler option.

I tried this code:
I can't share our internal code, but build something like serde_derive on windows and look for the proc macro dll. there should be a pdb next to it.
Next, you'll want to use ducible https://github.com/jasonwhite/ducible/ to remove a few other sources of non-determinism like timestamps.

ducible <dll> <pdb>

I expected to see this happen: Each clean build results in the same DLL and PDB (sha256 or similar).

Instead, this happened: the DLL is identical but the PDBs are not. Use something like https://www.cjmweb.net/vbindiff/ to spot the differences and the only one i noticed was this path to lib.def.

Meta

(also exists in latest master based on code inspection).
rustc --version --verbose:

rustc 1.42.0-nightly (212b2c7da 2020-01-30)
binary: rustc
commit-hash: 212b2c7da87f3086af535b33a9ca6b5242f2d5a7
commit-date: 2020-01-30
host: i686-pc-windows-msvc
release: 1.42.0-nightly
LLVM version: 9.0

Activity

added
O-windows-msvcToolchain: MSVC, Operating system: Windows
A-debuginfoArea: Debugging information in compiled programs (DWARF, PDB, etc.)
on Apr 30, 2020
added
A-reproducibilityArea: Reproducible / deterministic builds
T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.
on Apr 30, 2020
wesleywiser

wesleywiser commented on Jun 19, 2023

@wesleywiser
Member

Visited during wg-debugging triage. A number of other reproducibility issues are being looked at currently. Hopefully we can fix this soon. Marking P-medium.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-debuginfoArea: Debugging information in compiled programs (DWARF, PDB, etc.)A-reproducibilityArea: Reproducible / deterministic buildsC-bugCategory: This is a bug.O-windows-msvcToolchain: MSVC, Operating system: WindowsP-mediumMedium priorityT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @nikhilm@wesleywiser@jonas-schievink

        Issue actions

          Use of tempfile for lib.def leads to non-reproducible PDBs · Issue #71714 · rust-lang/rust