Reject extern "{abi}" when the target does not support it

[workingjubilee]

What

Promote unsupported_fn_ptr_calling_conventions from a warning to a hard error, making sure edge-cases will not escape. We now emit hard errors for every case we would return Invalid from AbiMap::canonize_abi during AST to HIR lowering. In particular, these architecture-specific ABIs now only compile on their architectures¹:

• amdgpu: "gpu-kernel"
• arm: "aapcs", "C-cmse-nonsecure-entry", "C-cmse-nonsecure-call"
• avr: "avr-interrupt", "avr-non-blocking-interrupt"
• msp430: "msp430-interrupt"
• nvptx64: "gpu-kernel", "ptx-kernel"
• riscv32 and riscv64: "riscv-interrupt-m", "riscv-interrupt-s"
• x86: "thiscall"
• x86 and x86_64: "x86-interrupt"
• x86_64: "sysv64", "win64"

The panoply of ABIs that are logically x86-specific but actually permitted on all Windows targets remain supported on Windows, as they were before. For non-Windows targets they error if the architecture does not match. This only causes using extern "{abi}" to have a consistent result without regard to its position in the grammar.

Moving the check into AST lowering is itself a breaking change in rare cases, above and beyond the cases rustc currently warns about. See "Why or Why Not" for details.

How

We modify rustc_ast_lowering to prevent unsupported ABIs from leaking through the HIR without being checked for target support. Previously ad-hoc checking on various HIR items required making sure we check every HIR item which could contain an extern "{abi}" string. This is a losing proposition compared to gating the lowering itself.

As a consequence, unsupported ABI strings will now hard-error instead of triggering the FCW unsupported_fn_ptr_calling_conventions. The code is also greatly simplified.

However, per #86232 this does cause errors for rare usages of extern "{abi}" that were theoretically possible to write in Rust source, without previous warning or error. For instance, trait declarations without impls were never checked. These are the exact kinds of leakages that this new approach prevents.

This differs from the following PRs:

• Add (back) unsupported_calling_conventions lint to reject more invalid calling conventions #141435 is orthogonal, as it adds a new lint for ABIs we have not warned on and are not touched by this PR
• Reject unsupported, unstable ABIs during AST lowering #141877 is subsumed by this, in that this simply cuts out bad functionality instead of adding epicycles for stable code

Why or Why Not

We already made the decision to issue the unsupported_fn_ptr_calling_conventions future compatibility warning. It has warned in dependencies since #135767, which reached stable with Rust 1.87. That was released on 2025 May 17, and it is now June. As we already had erred on these ABI strings in most other positions, and warn on stable for function pointer types, this breakage has had reasonable foreshadowing.

Upgrading the warning to an error addresses a real problem. In some cases the Rust compiler can attempt to actually compute the ABI for calling a function. We could accept this case and compute unsupported ABIs according to some other ABI, silently². However, this obviously exposes Rust to errors in codegen. We cannot lower directly to the "obvious", target-incorrect ABI and then trust code generators like LLVM to reliably error on these cases, either.

Refactoring the compiler so we could defer more ABI computations would be possible, but seems weakly motivated. Even if we succeeded, we would at minimum risk:

• exposing the "whack-a-mole" problem but "approaching linking" instead of "leaving AST"
• making it harder to reason about functions we can lower further
• complicating the compiler for no clear benefit

A deprecation cycle for the edge-cases could be implemented first, but it is not very useful for such marginal cases, like this trait declaration without a definition:

pub trait UsedToSneakBy {
    pub extern "gpu-kernel" fn sneaky();
}

Upon any impl, even for provided fn within trait declarations, e.g. pub extern "gpu-kernel" fn sneaky() {}, different HIR types were used which would, in fact, get checked. Likewise with anything with function pointers. Thus we would be discussing deprecation cycles for code that is impotent or forewarned³.

Implementing a deprecation cycle is possible, but it would likely require emitting multiple of a functionally identical warning or error on code that would not have multiple warnings or errors before. It is also not clear to me we would not find another, even more marginal edge-case that slipped through, as "things slip through" is the motivation for checking earlier. For this reason, any effort spent on additional warnings, as opposed to merely letting this PR languish for a bit, should require committing to limits in time and cost to user experience.

r? lang

Fixes #86232
Fixes #132430
Fixes #138738
Fixes #142107

Footnotes

1. Some already will not compile, due to reaching ICEs or LLVM errors. ↩

2. We already do this for all AbiStr we cannot parse, pretending they are ExternAbi::Rust, but we also emit an error to prevent reaching too far into codegen. ↩

3. It actually did appear in two cases in rustc's test suite because we are a collection of Rust edge-cases by the simple fact that we don't care if the code actually runs. These cases were excised in c1db989. ↩

[rustbot]

HIR ty lowering was modified

cc @fmease

This PR changes a file inside tests/crashes. If a crash was fixed, please move into the corresponding ui subdir and add 'Fixes #' to the PR description to autoclose the issue upon merge.

These commits modify tests/rustdoc-json.
rustdoc-json is a public (but unstable) interface.

Please ensure that if you've changed the output:

• It's intentional.
• The FORMAT_VERSION in src/librustdoc-json-types is bumped if necessary.

cc @aDotInTheVoid, @obi1kenobi

[workingjubilee]

@bors2 try

[rust-bors]

⌛ Trying commit c1db989 with merge 8b8eff5…

To cancel the try build, run the command @bors2 try cancel.

[rust-bors]

☀️ Try build successful (CI)
Build commit: 8b8eff5 (8b8eff55bd72abbb57167bc42222a7f91d41cb0d)

[workingjubilee]

@craterbot run mode=build-only name=pr-142134-abi-ast-error cap-lints=warn

[craterbot]

🚧 Experiment pr-142134-abi-ast-error is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

👌 Experiment pr-142134-abi-ast-error created and queued.
🤖 Automatically detected try build 8b8eff5
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🎉 Experiment pr-142134-abi-ast-error is completed!
📊 57 regressed and 16 fixed (643383 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[workingjubilee]

I'll send the necessary PRs to size-of and retour.

The primary culprit seems to be impls on function pointers, which are already warned on: https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=e1e8ca91878cbbcba5c6dc3b064cea2b

on retour

I should note that while impls on function pointers are convenient, it's not clear that code that works at the low level that retour does can be correct when interacting with these function pointers unless it is written with full awareness that the relevant ABIs are translated to extern "C". Some simple versions, maybe, but often detouring libraries have to make assumptions about codegen, and consistently getting that correct seems unlikely. So I believe enforcing this lint in a now-absolutist fashion will, in fact, make the world's code more correct.

on size_of

For size_of the upgrade to warn-on-deps seems to have been effective in getting the message across: Kixiron/size-of#6

No PRs or commits yet.

It also doesn't seem to have been updated for a fairly old feature name change, so it's broken for some features anyways, sadly: Kixiron/size-of#4

on the rest

• https://github.com/EnmanuelParache/msp430fr2476 is archived, but clearly only intended to be a crossbuild for msp430 devices which would support msp430-interrupt in the first place.
• https://github.com/ejaanens/msp430fr2476 is a skeleton project generated by svd2rust and has the same caveat.
• https://github.com/bekker/qvopenapi-rs looks abandoned and I expect a contrib will have a language barrier, but I'll try.

The "regressions" for libc, serde_derive, etc. all appear to be spurious failures due to build space running out.