Skip to content

compiletest: compare-mode cannot handle mixed success + failure #49855

Open
@pnkfelix

Description

@pnkfelix
Member

Sub-issue of #48879

Currently, if you have a ui test src/test/ui/foo.rs that signals a diagnostic error under the AST borrowck, but has no error under the NLL borrowck, then there is no way to make a foo.nll.stderr or foo.nll.stdout such that running compiletest --compare-mode=nll will accept it.

The reason: compiletest complains that the compilation itself was successful, and by default, all ui tests are assumed to signal a compilation failure.

The usual way such a case (of a test that compiles without an error) is handled in compiletest is to add a comment of the form // run-pass in the test itself. But we do not want the tests to be marked as always passing; they only pass under the special mode (NLL in this case) that is still under development in the compiler itself.

There are a number of reasonable ways of addressing this.

  1. We could modify such tests to ensure that they end with a function their fn main is tagged with #[rustc_error], so that they will never successfully compile.
  2. We could attempt to have some way to encode a mode-predicated run-pass marker, e.g. // [nll] run-pass.
  3. We could have compiletest infer that a test, when running under compare-mode, must have an implicit // run-pass marker, if there exists a foo.mode.stdout file (even an empty one).

I'm not such a big fan of options 1 or 2 because I like the fact that (so far) compare-mode does not have to impact the existing .rs nor .stderr/.stdout files; you just add your alternative .nll.stderr file and no one else is the wiser.

So currently I would like us to adopt option 3 (or some other option I haven't thought of yet) that allows the .rs/.stderr/.stdout to go on oblivious to the existence of compare-mode.

But, in the short term, to let us get work done, I plan to follow option 1 for all existing ui/ tests. (This is intended to be a temporary hack; all such uses of this hack should point back to this issue.)

Activity

added a commit that references this issue on Apr 10, 2018

Workaround rust-lang#49855 by forcing rustc_error in any mode, includ…

added a commit that references this issue on Apr 11, 2018

Auto merge of #49861 - pnkfelix:compare-mode-nll-followup-2, r=nikoma…

added
A-testsuiteArea: The testsuite used to check the correctness of rustc
C-enhancementCategory: An issue proposing an enhancement or a PR with one.
T-bootstrapRelevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
on Apr 11, 2018
pnkfelix

pnkfelix commented on Sep 14, 2018

@pnkfelix
MemberAuthor

GIven that the compiler team is talking about broader revisions to our test suites (e.g. turning them all into ui tests, and more consistently embedding meta-data like // run-pass into the headers of the tests themselves), I am now more inclined towards adopting option 2 above to resolve this in the long term:

  1. We could attempt to have some way to encode a mode-predicated run-pass marker, e.g. // [nll] run-pass.
steveklabnik

steveklabnik commented on Oct 21, 2019

@steveklabnik
Member

Triage; not aware of changes here, though this is a part of the repo that I'm not particularly familiar with.

moved this from Needs Triage / Backlog to Ready in compiletest maintenance and improvementson Jun 9, 2024
jieyouxu

jieyouxu commented on Oct 17, 2024

@jieyouxu
Member

compiletest triage: unclear what the current situation is, will need better understanding of compiler/types contributor workflows.

added
E-needs-designThis issue needs exploration and design to see how and if we can fix/implement it
E-needs-investigationCall for partcipation: This issues needs some investigation to determine current status
on Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-compiletestArea: The compiletest test runnerA-compiletest-compare-modesArea: compiletest compare-modesA-testsuiteArea: The testsuite used to check the correctness of rustcC-enhancementCategory: An issue proposing an enhancement or a PR with one.E-needs-designThis issue needs exploration and design to see how and if we can fix/implement itE-needs-investigationCall for partcipation: This issues needs some investigation to determine current statusT-bootstrapRelevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @steveklabnik@pnkfelix@pietroalbini@Mark-Simulacrum@jieyouxu

        Issue actions

          compiletest: compare-mode cannot handle mixed success + failure · Issue #49855 · rust-lang/rust