Proposal: Help search engines find latest version of a crate

[jsha]

Motivation and Summary

When someone uses a web search engine to find a crate’s documentation, they are likely to wind up on the documentation for a random older version of that crate. This can be confusing and frustrating if they are working on a project that depends on a more recent version of that crate. As an example: in April 2021, a Google search for [rustls serversession] links to version 0.5.5 of that crate, released Feb 2017. A Google search for [rustls clientsession] links to version 0.11.0, released Jan 2019. The latest version is 0.19.0, released Nov 2020.

To fix this, I propose that doc.rs’s URL structure should be more like crates.io: Each crate should have an unversioned URL (docs.rs/rustls/latest) that always shows the docs for the latest version of that crate. There would continue to be versioned URLs like today (https://docs.rs/rustls/0.19.0/rustls/), accessible as defined below. I believe this will, over time, lead search engines to more often find the unversioned URL.

This is a popular request:

https://github.com/rust-lang/docs.rs/issues/1006
https://github.com/rust-lang/docs.rs/issues/854
https://github.com/rust-lang/docs.rs/issues/74
https://github.com/rust-lang/docs.rs/issues/1411

It's also a problem that disproportionately affects new users of the language who haven't gotten in the habit of looking for the "go to latest version" link. I know when I was first learning Rust, this problem was a particular headache for me.

Non-working solutions

<link rel=canonical> is a commonly proposed solution, but it’s not the right fit:

The target (canonical) IRI MUST identify content that is either
duplicative or a superset of the content at the context (referring)
IRI.

Since documentation of different versions is not duplicative, this won’t work. And in fact search engines verify the property, and will disregard canonical links on a site if it does not hold.

Here are some links about Google’s handling of canonical:

https://developers.google.com/search/docs/advanced/guidelines/duplicate-content
https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls

Proposed Solution

For any given crate, https://docs.rs/<crate>/latest should exist and not be a redirect. It should serve the latest version of that crate. Crates.io should be updated so that the unversioned URL for a crate (e.g. https://crates.io/crates/ureq) links to the unversioned URL on docs.rs.

Sometimes people will want to link to a specific version of the documentation rather than the generic “latest” URL. There will be two ways to do that:

• You can navigate explicitly to the versioned URL, using the version selector tools already available in docs.rs.
• Additionally, we should follow GitHub’s UI for when you click on a line number in a source file. When you click on an anchor within a doc page, a little “...” should appear with options to either copy the line or copy a permalink. We may even want to mimic the keystroke to go to a permalink for a page (y).

Caching issues

Currently, only static files are cached. The things that change between versions of a crate are its HTML and some JS (containing JSON data used in displaying the pages). The HTML is currently not cached at all, so invalidating its cache is not a current concern. The JS is also not cached, but it has a unique URL per crate version so easily could be cached.

In case we later decide to start caching the HTML: The CloudFront rates for cache invalidation are reasonable: $0.005 per invalidation request, and purging a whole subdirectory (like /crates/<crate>/*) is considered a single invalidation request.

Will it work?

I’m pretty sure it will work. Search engines these days take navigation events heavily into account, so if most navigation events go to an unversioned URL, that will help a lot. Also, once we make this change, the unversioned URLs will start accumulating “link juice,” which will also help a lot.

One good demonstration that it will work is that crates.io already follows a scheme like this, and does not have the "links go to old versions" problem at all.

[pietroalbini]

The only reservation I have is the URL scheme: I'd prefer to have /<crate>/latest instead of /crates/<crate>, as it's consistent with what we have today and doesn't break the workflow of just typing docs.rs/crate-name in the URL bar. Otherwise I'm all for it!

[jsha]

Ah, yep, that URL scheme does make more sense for those reasons. Updated the proposal.

[jsha]

@rust-lang/docs-rs Does this approval sound good? From the above, @pietroalbini is for it but I'd like to get a little more buy-in from the team before I start work on it.

[syphar]

Hi @jsha , thanks for doing and pushing this!

I general I only have superficial knowledge of SEO / search engines, so really cannot judge on the best approach to optimize search results.

Still some thoughts:

• perhaps I'm missing something, but why don't we just index the latest version, and set rel=nofollow for older versions? Of course we could do this later if link juice alone doesn't help. Also, where would link juice come from if the results don't appear high enough in the search results?
• due to CSP we cannot cache the HTML anyway, so nothing we have to think about right now. But in general you're right, it's only about active invalidation at the right points. ( I have some ideas to solve this, but that would involve some more changes)
• where should our sitemap point to? Base-URL or /latest/?

[jsha]

perhaps I'm missing something, but why don't we just index the latest version, and set rel=nofollow for older versions?

A lot of the most important links, in terms of ranking, come from outside docs.rs. For instance, crates.io, GitHub, and pages discussing a crate. We don't control those directly so we can't stick rel=nofollow on links that go to older versions. Also, even if we could control them, we'd have to update all of them whenever a new version is released, to attach rel=nofollow to any links to the old version's URL.

Also, search engines no longer rank solely or even primarily based on links - real user navigations (measured various ways) count for a lot too. If users consistently navigate to the "latest" URL, that helps indicate that URL is more important.

Think of it this way: imagine each inbound link counts as 1 point and each navigation counts as 1 point. We currently have something like:

/foo/0.1.0: 5 points
/foo/0.2.0: 10 points
/foo/0.2.1: 2 points

We'd really rather have:

/foo/latest: 17 points

Also, where would link juice come from if the results don't appear high enough in the search results?

I'm not totally sure I understand the question, but I think it's: How will search engines find older crates? They'll be linked from https://docs.rs/crate/ureq/2.2.0.

where should our sitemap point to? Base-URL or /latest/?

/latest/

[jsha]

The change to /latest/ pages has been out for a while, and I still see Google returning old results for some queries that have good results available in /latest/. For instance, rustls streamowned returns links to 0.14.0 and 0.16.0, when 0.20.6 is the latest.

I now suspect there may be issues with crawling, where crates are often not crawled deeply enough because there are so many crates, and so many pages per crate. I suspect the Google Search Console would let me dig deeper and find that out. Would the docs.rs team be willing to authorize me to look at Search Console data for docs.rs? It would involve putting a TXT record with google-site-verification=XXXXXX in it or serving a specific HTML file at the root of the domain.

[jyn514]

from @pietroalbini:

I have no problem with giving them access to the webmaster console, but we should do it by having the infrastructure team setup the account and then granting jsha access to it, rather than having jsha setup their own account linked to docs.rs. I'll add that to my todo list.

[pietroalbini]

Done.

[jsha]

Thanks! Here's our coverage report: 1.26M valid URLs, 3.28M excluded. The top reasons are "Crawled - currently not indexed" and "Duplicate without user-selected canonical." Details on those reasons here: https://support.google.com/webmasters/answer/7440203#duplicate_page_without_canonical_tag

Drilling down, here is a sample of "Crawled - currently not indexed" URLs:

And "Duplicate without user-selected canonical":

We can drill down into individual URLs:

What I conclude from this first look is that we probably do need to implement rel="canonical". In its absence, Google is selecting a canonical page for us, and often getting it wrong, thus excluding the page we would like to have indexed. I'll make a proposal on #74.