Lang proposal: `extern "unspecified"` for naked functions with arbitrary ABI

[tgross35]

Background

One of the uses of naked functions is to implement custom calling conventions. We have some code in compiler-builtins like this:

// NOTE This function and the ones below are implemented using assembly because they are using a
// custom calling convention which can't be implemented using a normal Rust function.
#[unsafe(naked)]
pub unsafe extern "C" fn __aeabi_uidivmod() {
    core::arch::naked_asm!(
        "push {{lr}}",
        "sub sp, sp, #4",
        "mov r2, sp",
        "bl {trampoline}",
        "ldr r1, [sp]",
        "add sp, sp, #4",
        "pop {{pc}}",
        trampoline = sym crate::arm::__udivmodsi4
    );
}

The ABI needs to be specified, so extern "C" is used. However, this is misleading as the function does not actually use the C calling convention.

Correct ABI would be considered part of the preconditions for this function and it would only be callable inside an unsafe block, but Rust has no way to call the function correctly so it seems like we should prevent this.

Proposal

Add a new "unspecified" ABI that may be used with naked functions. Rust will error on attempts to call them.

/// # Safety
///
/// This function implements a custom calling convention that requires the
/// following inputs:
/// * `r8` contains a pointer
/// * `r9` contains a length
/// The pointer in `r8` must be valid for reads up to `r9` bytes.
///
/// `r8` and `r9` are clobbered but no other registers are.
#[unsafe(naked)]
pub unsafe extern "unspecified" fn foo() {
    core::arch::naked_asm!(
        // ...
    );
}

// SAFETY: `bar` is provided by `libbar.a` which we link.
unsafe extern "unspecified" {
    fn bar();
}

fn call_foo(buf: &[u8]) {
    // SAFETY: I didn't read the docs
    unsafe {
        foo();
        //~^ ERROR: `foo` has an unspecified ABI and cannot be called directly
        bar();
        //~^ ERROR: `bar` has an unspecified ABI and cannot be called directly
    }

    // SAFETY: call `foo` with its specified ABI, account for r8 & r9 clobbers
    unsafe {
        core::arch::asm!(
            "mov r8 {ptr}",
            "mov r9 {len}",
            "call {foo}",
            ptr = in(reg) buf.as_ptr(),
            len = in(reg) buf.len(),
            out("r8") _,
            out("r9") _,
            foo = sym foo,
        )
    }
}

Proposed rules:

1. extern "unspecified" can only be used with naked functions or extern blocks
2. The compiler will reject calling any functions marked extern "unspecified". It can still be passed as a function pointer, and it can be a sym in an asm block.
3. extern "unspecified" functions must be marked unsafe, and cannot be safe fn with an extern block. This is a hard error. (I'm less certain about this rule since unsafety doesn't mean much if you can't call it. Proposed because it seems consistent with how it must be used, given the function still has preconditions, and it's probably makes sense to treat them as unsafe in the compiler.)

Questions:

1. What should it be named? "unspecified", "none", "any", and "unknown" all seem workable. Also suggested in this thread: "custom", "uncallable".
2. Should parameters also be rejected? If the function is not callable, they don't serve much purpose other than documentation.

cc @rust-lang/lang, @folkertdev, @Amanieu

(currently empty) thread for discussion on Zulip: https://rust-lang.zulipchat.com/#narrow/channel/216763-project-inline-asm/topic/.60extern.20.22unspecified.22.60.20for.20naked.20functions/with/515596073

[folkertdev]

I really like this idea. I suspect there is also some embedded code that would make use of this, e.g.

https://github.com/rust-embedded/cortex-m/blob/c3d664bba1148cc2d0f963ebeb788aa347ba81f7/cortex-m-rt/src/lib.rs#L529

(this example came up here #140279 (comment))

extern "unspecified" functions must be marked unsafe, and cannot be safe fn with an extern block.

I think this is the right call, because of documentation and tooling. extern "unspecified" should still have a # Safety section with details on how C/assembly could call this function.

[traviscross]

Reading through it, this all sounds reasonable and right to me.

Maybe another name to consider for this ABI would be custom.

[traviscross]

I'll take assignment and "champion" this, as I seem to have been taking the asm ones recently. Under our process, with a champion, this can be implemented experimentally (and a tracking issue should be filed, etc.).

Of course, I'd first like to hear confirmation from @Amanieu that this seems reasonable. Also cc @RalfJung. And I'd like to hear from my fellow @rust-lang/lang members if anyone thinks this needs an RFC. My estimate is that it probably does not -- that this is a straightforward extension -- and that having a good stabilization report and the updated Reference documentation for this will be sufficient.

[workingjubilee]

There are other uses of an "uncallable-by-Rust-code" ABI that suggest considering a broader perspective.

[workingjubilee]

So, if the primary reason is to prevent Rust code from calling functions with that ABI, then there are already ~2 ABIs which need to have that implemented for them, and currently do not:

• extern "interrupt"
• extern "gpu-kernel"

extern "interrupt"

I say "two" and say extern "interrupt" instead of the half-dozen or so actual extern "???-interrupt" ABIs because those are not meaningfully different in terms of the actual semantics. All of the mentioned "ABIs" have an identically shared trait: they should not be callable by Rust code, and they restrict the arguments in their signature. In particular, almost all interrupt ABIs require the signature extern "interrupt" fn(), with the sole exception being extern "x86-interrupt"¹

extern "gpu-kernel"

AKA `extern "ptx-kernel", this has the same deal: appeared more target-specific than it actually is, as it interacts with a shared problem, and has the same detail of being a function that serves as a kind of entry point for a jump/branch/call, but one that cannot be Rust code compiled for that host. This is because what it actually exists is to represent an entry-point from a CPU to a GPU. Thus it can only be "valid to call" from the internals of the device driver, which can remotely move the GPU's program counter.

Lump? Split? Neither?

This is another instance of a classic "lumping vs. splitting" concern. I am not sure I strongly feel that the decision must cut one way or another. I do think that they have enough of a relationship that we will want to explicitly decide at some point in one direction or another. Call it an "unanswered question" or a "consideration for future directions" if you like?

• The case for lumping is that they are really all extern "uncallable" fn, in effect. Even that name is slightly off as they are all also "callable" in some sense, but only by a "magic" feat that is not available within the language per se.
• The case for splitting is that there may be enough other details that differentiate how certain things must be handled, and we do not expect them to be reasonable to handle by some other means like attributes on the functions in question.

Footnotes

1. and extern "x86-interrupt" has other severe drawbacks that recommend against ever actually using it, to the point it puts into question why we implement it. ↩

[asquared31415]

Personally I am in favor of the name unknown, because it's a calling convention not known to rust and therefore rust cannot directly call it. But bikeshedding aside, I really like this idea, and have had a few cases where it would have been useful, particularly in embedded and similar spaces with interacting with assembly or the system. I think that such an ABI is also suitable for the cases that Jubilee mentioned, even if they may not be perfect fits, it at least has the major component, which is that it cannot be called by rust, must be called externally ("externally" may be assembly or system interrupts or GPU or other means)

However, I think that a more refined solution for "this is a well defined ABI but must not be called by rust, it must be called by the GPU" would improve upon this.

Regarding processes, I think this is really only relevant to t-compiler, and maybe lang (it's technically syntax i guess?) and doesn't have any semantics, so I would expect this to just need an FCP to stabilize eventually.