Skip to content

doctests #5

Open
Open
@epage

Description

@epage

We talked about doctests being an area for improvement

Cases that don't work with doctests:

Activity

added
C-tracking-issueCategory: A tracking issue for something unstable.
S-needs-team-inputStatus: Needs input from team on whether/how to proceed.
on Feb 7, 2024
weihanglo

weihanglo commented on Feb 8, 2024

@weihanglo
Member

rustdoc is responsible for both compiling doc examples and executing them as tests. Cargo current has no way to control the behavior. There are a handful of issues around this:

Bonus

I propose to the team to looking into the possibility for rustdoc to delegate test executions to Cargo.

epage

epage commented on Feb 8, 2024

@epage
Author

If we can put as many doctests into a single binary as possible, that would also improve compilation times

We also want to make it so its as natural to get coverage for doctests as other tests.

I propose to the team to looking into the possibility for rustdoc to delegate test executions to Cargo.

That is the direction I would want to go as well.

weihanglo

weihanglo commented on Feb 8, 2024

@weihanglo
Member

Yeah we can push it further.

rustdoc, as a doc comments analyzer, provides source information via a flag. Cargo invokes rustdoc with the flag, so it knows where the source is and how to compile and test it.

GuillaumeGomez

GuillaumeGomez commented on Mar 21, 2024

@GuillaumeGomez
Member

I'm very fine with this approach as it would remove code from rustdoc. Problem is that the code examples originally were written to allow crate writers to add code examples for their users, meaning they needed to import the crate as if it was a dependency. I think this approach is still good, but should not be the only one (for example we can't run doctest in binary crates, which is bad). If cargo can solve both issues, that'd be awesome. :)

epage

epage commented on Mar 26, 2024

@epage
Author

Crazy idea: could we change the compilation of #[doc] to generate #[test] fns for each code fence.

  • If the test lib builds against the regular lib, then the import names still work
  • Otherwise, the doctests would be usable on private items, having access to everything where they are located.

Challenges

  • The compiler knowing enough about doctests to do this
  • The existence of compile tests (maybe an unstable feature to assert on compilation state?)
GuillaumeGomez

GuillaumeGomez commented on Mar 27, 2024

@GuillaumeGomez
Member

Unfortunately, it would require some new attributes for it to work as you can add a lot of different checks on doctests such as a specific edition or even to ensure that the compilation fails. More information here.

BurntSushi

BurntSushi commented on Mar 27, 2024

@BurntSushi
Member

Would it be possible to resolve how the doctest should be built with a new attribute? Maybe something like internal to opt into "compile this test as a normal unit test that uses crate internal APIs."

epage

epage commented on Mar 27, 2024

@epage
Author

Unfortunately, it would require some new attributes for it to work as you can add a lot of different checks on doctests such as a specific edition or even to ensure that the compilation fails

Yeah, I was aware of and called out compile_fail. Being able to set the edition is an odd one. Where can I read up more on the use case for that? As I don't personally have a use case for it, my automatic reaction is "can this be deprecated?" but I recognize there are likely cases I'm not aware of.

Likely switching to this style of doctests would require an opt-in with it being the default in a new edition, so we could get away with changing things.

epage

epage commented on Apr 23, 2024

@epage
Author

btw rust-lang/rust#123974 is a nice improvement to performance.

GuillaumeGomez

GuillaumeGomez commented on Apr 23, 2024

@GuillaumeGomez
Member

It's not finished yet. A few things need to be added:

  • only enable it starting 2024 edition
  • remove chunking to run all tests at once (and improve even further the performance improvement) and potentially fix libtest being blocked

Once done, I plan to work on binary doctests: I want to replace them with what we do for #[test] so people will be able to finally test their own code.

23 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-tracking-issueCategory: A tracking issue for something unstable.S-needs-team-inputStatus: Needs input from team on whether/how to proceed.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @epage@BurntSushi@GuillaumeGomez@weihanglo

        Issue actions

          doctests · Issue #5 · rust-lang/testing-devex-team