Allow setting RUSTC_BOOTSTRAP=1 on specific crates in the dependency graph from the top level of the build

[hsivonen]

The feature being requested

Please add a cargo capability to designate from the top level of the build which specific crates in the dependency graph are compiled with RUSTC_BOOTSTRAP=1 even when using a non-nightly compiler

Use cases

A piece of software that depends on many Rust crates (such as Firefox) tracks the stable release channel for the Rust compiler. Yet, in some case, it makes sense to use an unstable rust feature nonetheless.

1. There is a temporally substantial (not just a couple of Rust release trains but even years) mismatch between Rust stabilization priorities and the shipping needs of the piece of software that is using Rust code. The most obvious example of this is portable packed SIMD in Firefox.
2. A more temporary but immediate need comes up to use a Rust feature that has an active stabilization plan a couple release trains ahead of the Rust stabilization schedule. In the case of Firefox, some allocator-related customizations where of this type.

Note that making the first use case go away by stabilizing packed portable SIMD (which would be great!) does not make this feature request moot, since it seems reasonable to expect that the second kind of situation will keep arising from time to time even if there are periods in between when nightly features aren't needed.

Alternative solutions and what's wrong with them

Just use nightly Rust

The general tendency of Rust seems to be that if you want to use a nightly feature, you should be using nightly Rust. This poses problems:

1. If the piece of software using Rust is not only compiled for distribution by its developers but also by Linux and BSD distributions (as Firefox is), it is not scalable for each upstream project to designate its own snapshot of the Rust toolchain for the Linux and BSD distributions to use. In fact, for some distributions even the notion that building Firefox with in-archive compilers requires packaging the (Rust stable-channel) Rust toolchain on a rolling basis is a big departure from how they usually do things. Having upstream software require the packaging of upstream-designated snapshots of nightly Rust would probably not be well received.
2. Using nightly Rust means giving up on the value that the Rust release train model is supposed to provide. That is, the Rust release train model is supposed to catch regressions that land in nightly before they reach the stable channel. If there truly is such quality assurance value in Rust's train model, it seems bad to ask projects to give up that value for all code they compile if in a vast amount of code they have a couple of specific things that need nightly-only features.
3. Especially given the first point, "just use nightly Rust" does not really work as a temporary solution to a temporary problem (like the allocator-related customization in Firefox), because in the context of downstream packaging, the timescales to plan for such a change (if feasible at all; see the first point) would likely exceed the time period when a given nightly feature is temporarily needed. Notably, "just use nightly Rustis not just a quick matter ofrustup default nightly` for software that gets packaged downstream.

Additionally, the "just use nighly Rust" solution has the problems of the next solution as well.

Set RUSTC_BOOTSTRAP=1 for the whole build system

This is what Firefox used to do.

The main problem with this solution is that it's very easy to casually introduce more dependencies on nightly Rust features than was intended. Specifically, if nightly features are enabled for specific purpose that's well-known to people who take care of toolchain issues in the project, there is some chance of tracking the status of each nightly feature of interest in Rust development. If, however, developers casually introduce more dependencies on nightly features, soon no one is tracking which nightly features the project as a whole depends on ending there may be unpleasant surprises if nightly feature breaks and the person who casually started using it hasn't been tracking the status of the feature and hasn't been prepared to deal with the nightly feature changing.

In general, to the extent it is some sort of Rust position that production users of Rust should not be relying on nightly features, it seems incongruent with that position to suggest solutions that easily lead production users to depend on arbitrary nightly features when their use case is about depending on a very small number of them.

Notably, unintentionally adding a dependency on a nightly feature is not necessarily a matter of a developer "unintentionally" typing #![feature(foo)] in the code they are writing: #![feature(foo)] may be buried in a dependency, and with nightly features being behind cargo features in dependencies, it may be even less obvious what nightly features are enabled.

Use println!("cargo:rustc-env=RUSTC_BOOTSTRAP=1"); from build.rs

This is what Firefox does now.

This hack avoids the problems of the above two solutions by scoping RUSTC_BOOTSTRAP=1 to specific crates. It's own problems fundamentally arise from the hack being syntactically part of an individual crate while the the need for the hack is a configuration concern of the top-level application.

In a non-cargo vendor scenario, putting application-level configuration in a specific crate basically requires maintaining a git fork the crate with an extra change set just for the configuration and the ceremony of telling cargo to replace the crates.io instance of a crate with the forked git instance. Even in a cargo vendor situation is not a matter of maintaining a patch that always gets applied after re-running cargo vendor: cargo requires more ceremony to opt out of hash checking for the vendor crates.

Even if the effort required to maintain application-level configuration as a change to an individual crate is relatively mild, the path of least maintenance burden still is to make the change in the crates.io crate itself. This poses the risk of other applications unknowingly becoming reliant on configuration meant for a different application. This is probably harmless for crates that are blatantly obviously nightly-only, such as previously the simd crate or presently the packed_simd crate. It can be problematic, however, on crates that conditionally depend on such crates. For example, encoding_rs has a cargo feature simd-accel for enabling the dependency on, depending on encoding_rs version, simd or packed_simd and gaining the associated performance benefits. While it should be obvious that this is a cargo feature as opposed to the default state of things precisely because there is some tradeoff (in this case, trading compatibility with future Rust toolchains for performance today), the full implications of the trade-off might not be obvious and someone might enable simd-accel with a stable-channel toolchain and be unhappy later when it toolchain update breaks the build.

Preferred solution

My preference for solving this would be adding the ability to use the top-level Cargo.toml to indicate which crates in the dependency graph are to be compiled with nightly features enabled. While naming is a total bikeshed, I suggest putting something very obvious in the syntax along the lines of enable_features_that_future_rust_releases_may_break instead of relying on short but less obvious terms like nightly or bootstrap.

There is other per-crate configuration tha the top-level Cargo.toml should be able to control, such as the optimization level. In principle, it would be good for the solution for this feature request to be consistent with other such configuration, but I also wish that this feature request doesn't get deferred indefinitely due to consistency concerns.

Alternative solution

It has been suggested that instead of introducing syntax into the the top-level Cargo.toml the RUSTC_BOOTSTRAP environment variable should be extended to take a list of crate names as an alternative to taking the value 1. While this would address the use case, I consider it better design to recognize the use case and to provide top-level Cargo.toml syntax (that makes the downsides clear) along with other per-crate configuration than to make RUSTC_BOOTSTRAP more magic with a special-case design that sort of tries to avoid recognizing the use case by deliberately keeping the mechanism obscure.

Potential downsides

A potential downside is that people use enable_features_that_future_rust_releases_may_break and are still unhappy when breakage happens. However, the standard approach of telling people to "just use nightly Rust" already has this problem.

The best solution to avoiding unhappiness from changes to features that are explicitly subject to change is to work actively on "subject to change" features so that they actually don't appear de facto unchanging and so that they spend a relatively short time in the "subject to change" state. Fixing that kind of general prioritization issue is outside the scope of this feature request, though.

[nox]

While this would address the use case, I consider it better design to recognize the use case and to provide top-level Cargo.toml syntax (that makes the downsides clear) along with other per-crate configuration than to make RUSTC_BOOTSTRAP more magic with a special-case design that sort of tries to avoid recognizing the use case by deliberately keeping the mechanism obscure.

That’s a feature. I’m against anything that lets you set RUSTC_BOOTSTRAP from within Cargo itself. Your own mach build tool should be setting it on the command-line to Cargo.

I’m fine with a comma-separated list of crate names in RUSTC_BOOTSTRAP but my final opinion is that this is an anti-use case and that Firefox in the first place should just be using nightly if it wants nightly features.

[nox]

The main problem with this solution is that it's very easy to casually introduce more dependencies on nightly Rust features than was intended. Specifically, if nightly features are enabled for specific purpose that's well-known to people who take care of toolchain issues in the project, there is some chance of tracking the status of each nightly feature of interest in Rust development. If, however, developers casually introduce more dependencies on nightly features, soon no one is tracking which nightly features the project as a whole depends on ending there may be unpleasant surprises if nightly feature breaks and the person who casually started using it hasn't been tracking the status of the feature and hasn't been prepared to deal with the nightly feature changing.

For the record, it has never happened in Servo that we would just accidently start using a nightly feature, and AFAIK it never happened in Gecko either before you started using RUSTC_BOOTSTRAP on build scripts.

[hsivonen]

I’m against anything that lets you set RUSTC_BOOTSTRAP from within Cargo itself. Your own mach build tool should be setting it on the command-line to Cargo.

I’m fine with a comma-separated list of crate names in RUSTC_BOOTSTRAP

What problem are you seeking to solve by having mach communicate a list of crate names via an environment variable compared to the feature being fully contained in cargo and controllable from the top-level Cargo.toml?

my final opinion

I guess that means I should not try to change your mind, but I'm still interested in understanding your rationale.

Firefox in the first place should just be using nightly if it wants nightly features

This suggests that you are not against using nightly features in production. If you are not against using nightly features in production, why is it important not to use them with a non-nightly compiler?

My clone of Gecko, which is a bit out of date, has 385 crates, though some of them are not code that ships in the product. Currently, 2 of the crates need nightly features. A few months back, the number was slightly larger, but still on the order of "up to 5".

Why is it important to deny the QA benefits of the Rust train model and the distro packaging clarity to all these crates if a couple of them use nightly features? What's the point of keeping stable Rust so pure as not to use it?

If Firefox were to use nightly Rust, what do you expect would happen to Linux and BSD distros packaging Firefox and Rust? How would what you'd expect to happen be an improvement compared to what this feature request contemplates?

[sfackler]

What's the point of keeping stable Rust so pure as not to use it?

Just so I understand, you're confused as to why we would want to prohibit a stable compiler from using unstable features?

You are not bootstrapping rustc. Don't use RUSTC_BOOTSTRAP.

[petrochenkov]

@sfackler
Perhaps a slight rewording will help to look at this situation from another point.

Firefox doesn't want to use a "stable compiler", Firefox wants to use a "pinned nightly compiler" that happens to be pinned to exactly the same commit as stable release.
That's probably the most reasonable pinned nightly to use, for multiple reasons.
(And it doesn't undermine any stability if you formulate it like this.)

The problem is that RUST_BOOTSTRAP is the least painful way to do it, despite not being related to bootstrapping rustc.

[dwijnand]

Shouldn't production Firefox use production Rust?

[hsivonen]

(And it doesn't undermine any stability if you formulate it like this.)

The problem is that RUST_BOOTSTRAP is the least painful way to do it, despite not being related to bootstrapping rustc.

That's why I suggested naming along the lines of enable_features_that_future_rust_releases_may_break without the word "bootstrap".

(It seems relevant that RUSTC_BOOTSTRAP's current state was motivated by distro use cases. This could be considered another distro-relevant use case, since alternatives to Firefox using RUSTC_BOOTSTRAP would need to answer the question: "How would distro packaging of Firefox work?")

[skade]

@petrochenkov If you want to use stable as a pinned nightly, set RUSTC_BOOTSTRAP globally. What is requested here is an extension actively catering to that usecase.

I maintain my point made in the other issues that this is wilfully trying to tear down guarantees that we have worked hard to set up.

Also, I'd like to see examples outside of Firefox where users have complained that a nightly feature broke while they had opted into using nightly. From my experience in the embedded area, people appreciated the clarity and were willing to opt into the breakage.

[sfackler]

I am actively opposed to make it easier to use RUSTC_BOOTSTRAP.

[hsivonen]

For the record, it has never happened in Servo that we would just accidently start using a nightly feature, and AFAIK it never happened in Gecko either before you started using RUSTC_BOOTSTRAP on build scripts.

@glandium recalled otherwise in his earlier comment on your PR. (I don't know specifics either way.)

I maintain my point made in the other issues that this is wilfully trying to tear down guarantees that we have worked hard to set up.

I suggest approaching this from the point of view of people trying address use cases under constraints rather than people "willfully trying to tear down". (If you've designed a system and people are "holding it wrong" on purpose, it's a good idea to consider how the system isn't meeting their needs as designed.)

I reiterate that Firefox used to have nightly Rust features enabled globally and then moved to more granular containment of nightly features. My understanding of the motivation of the change was the opposite of "trying to tear down" Rust's guarantees.

I have a hard time understanding how it would be better that what this issues requests, but if the Rust team would be happier with Firefox enabling RUSTC_BOOTSTRAP=1 globally, I could live with that, but I'm not a Firefox build system developer.

(For clarity: I'm not a Firefox build system developer. My interest is being able to conform to social norms of crates.io while having a couple of crates build in the context of Firefox with nightly features with minimal need to maintain forks of the crates until packed_simd becomes std::simd.)

Also, I'd like to see examples outside of Firefox where users have complained that a nightly feature broke while they had opted into using nightly.

That can be read is implying that in the case of Firefox someone complained about a nightly feature breaking. Did you mean that? If so, what was the complaint?

From my experience in the embedded area, people appreciated the clarity and were willing to opt into the breakage.

I note that this feature request is formulated to show clarity of breakage being opt-in. The question is one of granularity of the opt-in.

I am actively opposed to make it easier to use RUSTC_BOOTSTRAP.

This statement doesn't explain why and doesn't indicate how you suggest the relevant use cases be addressed.

[nox]

I note that this feature request is formulated to show clarity of breakage being opt-in. The question is one of granularity of the opt-in.

This feature request is to make crates pretend they build on stable while they in reality use nightly features. That by definition tears down Rust’s guarantees.

[dwijnand]

My interest is being able to conform to social norms of crates.io while having a couple of crates build in the context of Firefox with nightly features with minimal need to maintain forks of the crates until packed_simd becomes std::simd.

I think the problem is that cargo builds both end-user apps, like Firefox, and crates. So while it's valid in the context of Firefox's build it's dangerous for the rest of the ecosystem.

[hsivonen]

This feature request is to make crates pretend they build on stable while they in reality use nightly features.

No, the opposite. This is about enabling configuration of whether nightly features are enabled on a per-crate basis but not from the crates themselves—only from the top-level Cargo.toml.