Failed to compile a project with LTO + PGO

[zamazan4ik]

In vectordotdev/vector#15631 (comment) I tried to optimize Vector with LTO + PGO enabled. But during the optimization phase (done with cargo pgo optimize) I get the following error: Kobzol/cargo-pgo#32 (comment) . I am not the only person with this error - llvm/llvm-project#57501 (comment) (@kevincox probably can provide more details about they project).

The only way to avoid the bug is disable LTO completely (switching from Fat to Thin mode does not help).

I expected to see this happen: Vector compiles successfully with LTO and PGO enabled.

Instead, this happened: Vector failed to compile with LTO + PGO.

Meta

rustc --version --verbose (from https://github.com/vectordotdev/vector/blob/master/rust-toolchain.toml):

1.71.1

Other links:

• Possibly related bug in LLVM (but I am not sure): [15 regression] "module flag identifiers must be unique (or of 'require' type) llvm/llvm-project#57501
• A discussion in cargo-pgo: LLVM ERROR: Broken module found, compilation aborted Kobzol/cargo-pgo#32

[Kobzol]

Were you able to reproduce this without using cargo pgo, by using the compiler flags manually? (-Cprofile-generate etc.)

[zamazan4ik]

I haven't tried to do it yet but I will (and then report the results here)

[kevincox]

My project was done using the flags directly. See https://gitlab.com/kevincox/ricochetrobots/-/blob/6bbc44ddbf2ab8ae9edcbdca62e3035572f9c1ed/default.nix#L41-57

# Cargo.toml
[profile.release]
codegen-units = 1
lto = true

RUSTFLAGS=-Cprofile-use={profile-data}

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ricochetrobots` (bin "ricochetrobots-debug-precompute")
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ricochetrobots` (bin "solve")
[naersk] cargo returned with exit code 101, exiting

[caizixian]

I'm using the compiler flag directly (via RUSTFLAGS). RUSTFLAGS="-Cprofile-use=/tmp/pgo-data/merged.profdata -Cllvm-args=-pgo-warn-missing-function" fails with

error: file `/tmp/pgo-data/merged.profdata` passed to `-C profile-use` does not exist.`

The file is confirmed to exist via ls. The merged.profdata is generated using llvm-profdata shipped with Rust 1.71.1.

RUSTFLAGS="-Clto=false -Cprofile-use=/tmp/pgo-data/merged.profdata -Cllvm-args=-pgo-warn-missing-function" works.

My PGO+LTO setup worked with Rust 1.66.1, but doesn't work with 1.71.1.

[lqd]

This "file does not exist" error seems like a different one from the OP ("LLVM ERROR: Broken module found, compilation aborted!"), and it would be good to have a small self-contained example for your new error -- and possibly a dedicated issue since it also looks different from #63609. It feels possible to investigate in mmtk-openjdk directly, but to land a fix with a non-regression test, a small reproducer (MCVE) is going to be important.

[zamazan4ik]

I got the same issue during the ruff optimization with PGO with cargo-pgo. When I run cargo pgo optimize bench in https://github.com/astral-sh/ruff/tree/main/crates/ruff_benchmark on my Mac machine (Macbook M1 Pro, macOS 13.4 Ventura), I get the following error:

<skipped lines>
Compiling ruff_benchmark v0.0.0 (/Users/zamazan4ik/open_source/ruff/crates/ruff_benchmark)
warning: 19 warnings emitted========>  ] 287/299: ruff_benchmark, libcst, similar, ruff_notebook, ruff_python_parser, pyproject-toml, ruff_python_formatter, criterion

warning: 468 warnings emitted========> ] 290/299: libcst, ruff_notebook, ruff_python_parser, ruff_python_formatter, criterion

warning: 996 warnings emitted========> ] 292/299: libcst, ruff_python_parser, ruff_python_formatter

   Compiling ruff v0.0.287 (/Users/zamazan4ik/open_source/ruff/crates/ruff)
warning: 75 warnings emitted=========> ] 294/299: ruff, ruff_python_parser

warning: 5188 warnings emitted=======> ] 295/299: ruff

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ruff_benchmark` (bench "formatter"); 45 warnings emitted
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ruff_benchmark` (bench "parser"); 42 warnings emitted
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ruff_benchmark` (bench "linter"); 47 warnings emitted
PGO optimized build has failed.
Cargo finished with an error (101)

Note that Ruff also enables LTO as a part of its Release profile. When I switch off LTO, cargo pgo optimize bench builds Ruff successfully.

Additional details - cargo pgo info:

cargo pgo info
[rustc version]: 1.72.0 is recent enough
[llvm-profdata]: found at /Users/zamazan4ik/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/llvm-profdata
[llvm-bolt]: could not be found (Build LLVM with BOLT and add its `bin` directory to PATH.)
[merge-fdata]: could not be found (Build LLVM with BOLT and add its `bin` directory to PATH.)
Some requirements were not satisfied

Tested the same scenario on my Linux machine (Fedora 38, x86-64) for the same scenario with the same Rust version and got the same error.