rustdoc fails unpleasantly when it can't tokenise a code block

[chris-morgan]

Consider a file like this:

#[crate_id = "sad"];

/// ```
/// a + b ∈ Self ∀ a, b ∈ Self
/// ```
pub fn foo() {}

Here's rustdoc's opinion of this:

<stdin>:1:7: 1:8 error: unknown start of token: \u2208
<stdin>:1 a + b ∈ Self ∀ a, b ∈ Self
                ^
task '<unnamed>' failed at '~Any', /home/chris/vc/rust/src/libsyntax/diagnostic.rs:48

☹

You see, it attempted to tokenise the contents of that code block.

The current solution for this is marking that code block as "notrust":

/// ```notrust
/// a + b ∈ Self ∀ a, b ∈ Self
/// ```
pub fn foo() {}

rustdoc should do one of two things:

1. Permit tokenisation to fail silently, discarding the notion of syntax highlighting should it fail; or

2. Retain the current notrust-must-be-explicit behaviour, but with two alterations:

   1. Give meaningful spans, if at all possible—sad.rs:4:11: 4:12 as the error span rather than <stdin>:1:7: 1:7.
   2. Give you a hint that that might be want you meant.

   In total, perhaps something like this:

   <stdin>:1:7: 1:8 error: unknown start of token: \u2208
   <stdin>:1 a + b ∈ Self ∀ a, b ∈ Self
                   ^
   sad.rs:3:5: 3:8 info: that code block does not tokenise as Rust code; perhaps you meant to mark it as notrust?
                         (e.g. use the opening fence "```notrust")
   sad.rs:3 /// ```
                ^~~
   

[dennisoh]

I think that "A-rustdoc" label can be added.

[steveklabnik]

Triage: no changes.

[frewsxcv]

Relevant issue: servo/servo#9704