Migrate from hoedown to pulldown-cmark for markdown parsing

[frewsxcv]

For markdown parsing, we currently use rust-lang/hoedown (based on hoedown/hoedown). There's a (somewhat) new Markdown pull parser entirely written in Rust: google/pulldown-cmark. Some reasons to switch:

• All the safety guarantees that come with anything written in Rust
• Rust contributors don't need to know C if they want to make changes upstream
• Don't need to maintain our own C hoedown bindings / pulldown-cmark has an idiomatic Rust API
• It uses Cargo, which should make integration easy

[frewsxcv]

cc @rust-lang/tools @rust-lang/docs

@GuillaumeGomez has already expressed interest in making the changes for this transition.

[frewsxcv]

Via #29329 and critiqjo/rustdoc#2, it looks like @critiqjo has already made some progress on migrating to pulldown-cmark.

[alexcrichton]

We're not quite ready to pull in crates.io dependencies into the build system yet as we've still got makefiles that don't work with that. On 2016-02-02, however, I'll be deleting the makefiles with glee at which point we are capable of pulling in dependencies from crates.io!

[frewsxcv]

2016-02-02

I'm going to guess you meant 2017 here

[alexcrichton]

Oops, yes!

[steveklabnik]

I have been very interested in this for a long time.

[matklad]

Some random thoughts:

1. Are these two implementations compatible? Probably not, and that means some docs might break a little bit.

2. Perhaps this is the chance to specify what exactly is the markdown dialect of the docs?

3. What's the current plan about using ::foo::bar::baz syntax for cross-link the docs? If we decide to add some kind of linking, will it be compatible with pulldown-mark?

[frewsxcv]

Are these two implementations compatible? Probably not, and that means some docs might break a little bit.

pulldown-cmark is an implementation of CommonMark.

Hoedown follows the original markdown document, but there are talks about switching to CommonMark.

From what I understand, one of the main reason CommonMark exists is because the original Markdown specification is so vague and there were many subtle differences in implementations. More info in this thread. I'm not familiar enough with either spec to know specific differences.

What's the current plan about using::foo::bar::baz syntax for cross-link the docs? If we decide to add some kind of linking, will it be compatible with pulldown-mark?

This is what led me to open this issue. There aren't any concrete plans right now for this syntax, but if this something that is decided cmark-pulldown can help. There are talks about adding a basic generic directives/plugins syntax into CommonMark which would be great since we wouldn't need to invent our own syntax. Though, if we did invent our own syntax, from what I've seen of the pulldown-cmark API, it seems it would be easier to implement in that than Hoedown.