Proposal: Convert the `typing`-module PEPs into a series of HOWTOs

[AlexWaygood]

Problem description

The typing module docs have a problem.

typing.rst is littered with references to the many typing-related PEPs that have been accepted by the Steering Council. Often these references to PEPs are used in lieu of detailed examples or explanation as to how a feature is supposed to work.

However, PEPs are not supposed to be living pieces of documentation. As such, proposals to update old typing PEPs to reflect changes in implementation details have been rejected. But when these PEPs are the closest thing we have to a full Python-typing specification, this means that our specifications are slowly growing out of date.

This unfortunate situation has also been recognised by the Steering Council in a recent email to the typing-sig mailing list.

Proposed solution

I propose that we slowly start to convert various typing PEPs into a series of Python-typing HOWTOs. In this way, we'd follow the example of other modules such as enum or socket. These modules have terse API references in socket.rst and enum.rst, but these API references are supplemented with expansive HOWTOs that serve as a guide for users who are new to the respective modules. Given the number of typing PEPs, the scale of the task for typing will be larger; but I still believe that this is a good example to follow.

Specifically, I think we should have something like the following structure:

1. "Introduction to type hints in Python". This HOWTO would combine material from the following sources (updating examples where relevant to use modern syntax such as PEP-604 union types).
   1. PEPs 483 & 484 (introducing type hints to Python)
   2. PEP 526 (syntax for variable annotations, ClassVar)
   3. PEP 613 (explicit type aliases)
   4. PEP 673 (typing.Self)
   5. The sections in typing.rst on Any, Callable and type aliases.
   6. Some of the material in typing.rst on TypeVar and generics (but only the most basic material).
2. "Generics HOWTO". While the intro HOWTO would introduce the basic concept of TypeVars, this HOWTO would tackle the nitty-gritty:
   1. Using typing.Generic to make generic classes.
   2. The differences between bound TypeVars, constrained TypeVars, and TypeVars which are neither.
   3. Covariance versus contravariance versus invariance.
   4. Maybe a brief mention of PEP 585 (generic stdlib collections).
   5. ParamSpec and Concatenate (material from PEP 612).
   6. Variadic generics (material from PEP 646).
3. "Structural subtyping HOWTO", using material from PEP 544 and the section in typing.rst on structural subtyping and typing.Protocol.
4. "TypedDict HOWTO", using material from PEPs 589 & 655, as well as the section on TypedDict in typing.rst.
5. "Literal types HOWTO", describing how to use PEPs 586 (typing.Literal) & 675 (typing.LiteralString).

Due to the brevity of their PEPs, and the fact that they don't really "fit in" neatly with any of the other proposed HOWTOs, I propose that the following typing features have their full specifications integrated into the typing.rst API reference, rather than having dedicated HOWTOs:

1. "Final/@final (PEP 581)
2. Annotated (PEP 593)
3. TypeGuard (PEP 647)

Why not have this information at typing.readthedocs.io?

There's an ongoing effort, spearheaded by @srittau among others, to include a body of typing documentation at https://typing.readthedocs.io/en/latest/. I am fully in support of this effort, and see the documentation there as complementary to the documentation that currently lives in the typing PEPs. However, I do not believe that the documentation currently in the typing PEPs should belong on that site, for several reasons:

• https://typing.readthedocs.io/en/latest/ is a third-party site, rather than being an "official" corpus of Python-language documentation. This is a good thing -- it allows contributors to that site to iterate and improve on the documentation there at a more rapid pace. However, the information in the typing PEPs are officially recognised changes to the language and the language's typing framework. As such, -- in my opinion -- it should live in the central docs.python.org documentation along with the documentation for the rest of the language.
• https://typing.readthedocs.io/en/latest/ is currently not particularly discoverable via search engines. For example, it does not appear in the first page of Google results for "Python typing".

Questions to be discussed

1. What do we think of the general idea of moving material out of PEPs, providing specifications through a series of HOWTOs, and slimming down typing.rst into a terser API reference?
2. What do we think of my specific proposal for how to structure this series of HOWTOs?

Copied to:

• typing module maintainers: @gvanrossum, @Fidget-Spinner, @JelleZijlstra
• @srittau, who does a lot of valuable work on typing documentation over at the https://github.com/python/typing repo.
• @hauntsaninja, who recently expressed interest in helping with typing docs.
• Several miscellaneous typing-PEP authors who may be interested in this proposal: @ambv, @JukkaL, @ilevkivskyi, @mrkmndz, @pradeep90, @Gobot1234, @mrahtz, @davidfstr, @erictraut
• @encukou, who recently sent the email to typing-sig on this topic on behalf of the Steering Council, and is also a member of the docs working group.

Linked PRs

• gh-91533: Move much of typing.rst to a new 'Guide to typing features' document #105827

[AlexWaygood]

(In case it's not obvious: I'm happy to invest time and energy into writing some of these, if we agree that this is a good solution to the problem.)

[srittau]

I would invite you to have this discussion over at https://github.com/python/typing/issues. The documentation over there (which is currently published to https://typing.readthedocs.io) is intended to be the comprehensive source for typing related documentation. There already are a few issues about a possible structure on that site. Fragmenting the existing documentation even more should not be the goal.

readthedocs was chosen for an easy to set-up documentation target. Eventually integrating the documentation with docs.python.org should be a goal, though.

[srittau]

Also cc @shannonzhu who set up a lot of the infrastructure of the typing docs.

[AlexWaygood]

I would invite you to have this discussion over at https://github.com/python/typing/issues.

The core issue, in my opinion, is that the typing docs at https://docs.python.org/3/library/typing.html currently link far too much to typing PEPs. These are not meant to be living pieces of documentation, and so are rapidly becoming outdated. This is an issue with the typing docs living in the CPython repo, so I think it makes sense to have this conversation in the CPython issue tracker.

One possible alternative solution to the problem, as you say, is to invest energy into further expanding the documentation at https://typing.readthedocs.io/, and link heavily to that corpus of documentation from the docs at https://docs.python.org/3/library/typing.html.

[encukou]

This sounds great to me!

One argument for typing.readthedocs.io is that a lot of the information is not tied to Python versions, or is usable from third-party backport modules.
It's fine to mention typing-extensions in Python docs, but keeping the docs updated across Python versions might be too much of a hassle. And typing.readthedocs.io could have things like proposed extensions, tool compatibility overviews.
IMO, typing.readthedocs.io can be made official enough by linking to it from docs.python.org.
Maybe a good process would be to draft articles on RTD and move to HOWTOs when finalized?

(I'm just throwing ideas around, not arguing a position. Please consider them, but you don't need to convince me.)

[JelleZijlstra]

Thanks for starting this effort! I'd been thinking about working on this area too. In my mind, there are several different goals we could work towards:

1. A comprehensive spec that replaces the PEPs. Type checkers would work towards implementing this spec, and the typing community would make small changes based on consensus. We could replace links in the docs with these.
2. Detailed end-user documentation on how to use typing.

These two perhaps shouldn't be the same document. A spec is useful for advanced users, but for most users it's not the right place to start. I'd want to focus on the spec, for a couple of reasons: it's easier since we can mostly glue together the spec sections of the PEPs; it allows us to change the spec in the future (which we sometimes want to do); and there are already excellent documentation resources online (e.g., the mypy docs).

So here's what I'd want to see:

• We create a comprehensive typing spec, initially created by gluing together the text from all the relevant PEPs
• We link to existing online resources (notably the mypy docs) where possible.
• The typing module documentation should focus on listing the names available in the typing module and giving brief introductions to each.

I don't feel too strongly about where to put the spec or docs. typing.readthedocs.io is nice but we haven't made much progress with it and it's true that it may be less discoverable.

[AlexWaygood]

I'll cross-reference the related issue over at the python/typing repo, so that people can have a more full consideration of the (slightly competing, but I think/hope also somewhat complementary 🙂) proposals:

• Document Structure and Wishlist typing#845

[AlexWaygood]

We link to existing online resources (notably the mypy docs) where possible.

+1 for looking to the mypy docs as an example -- they're a really impressive body of typing documentation. I'd be wary of linking to them too heavily, though, as they are also becoming outdated in places -- we'd effectively be relying on a third-party project having well-maintained documentation. mypy also can't document any typing features that they don't yet support, whereas CPython should document the latest typing features, even if they are not yet supported by all type-checkers (and full support of new typing features by third-party type checkers can often take >1 year).

[gvanrossum]

Would it make sense to kick-start the typing docs for end users by selectively copying sections from the mypy docs (with some kind of attribution)? Just like starting the specs with the existing PEPs.

[AlexWaygood]

I guess I'm not 100% sold on the distinction between specifications and end-user documentation. I get the idea, but I worry that we'll end up with duplication between the different pieces of documentation. For example, PEP 544 is mostly a specification. But I've also seen it remarked upon in several places (e.g. here, or here) that it serves as an excellent piece of end-user documentation, which is better than anything currently at https://docs.python.org/3/library/typing.html.

[davidfstr]

I agree that it would be useful to collect a set of official “typing Guides/HOWTOs”, which are distinct from the typing-related PEPs (which are deltas applied on top of each other).

I agree that the mypy documentation would be a good starting point for this effort. This is the documentation that I currently consult first for guide-level typing documentation.

I don’t particularly care where such guide level documentation lives. I do think, however that it might be best suited to be in a site external from the versioned Python documentation, since type annotations are mostly agnostic to whatever Python version is in use. (In particular you can use very new typing features in older versions of Python.) I do think, however that typing.rst from the Python documentation should include prominent links to this external site (at least to the site’s top-level) to make it discoverable (both by users and search engines).

I am aware of the existing https://typing.readthedocs.io/en/latest/ site. A cursory reading of that site’s current content feels it is currently used to house detailed typing related documentation that didn’t already make it into a PEP or somewhere else (like the mypy documentation). For example it details the handling of __all__ across current-generation type checkers. In particular I do not currently see any entry-level guides like “Introduction to optional static typing in Python” or “Why should I care about type annotations in Python?” on that site at this time.

It would make sense to me to focus this documentation effort on improving the content at the existing https://typing.readthedocs.io/en/latest/ site, and linking it to from the official versioned Python documentation when the site is deemed “ready” (for some definition of “ready”).

Also, I like the organization of guides proposed by the originator of this thread. ( @AlexWaygood )

[AlexWaygood]

It sounds like the prevailing mood is to invest energy into improving the corpus of documentation at typing.readhedocs.io, and then — once they're ready — link heavily from typing.rst in the CPython docs to that body of documentation. Others in this thread have made persuasive arguments as to why that might be preferable, so I'll happily concede that point.

I still have some concerns about the discoverability of typing.readhedocs.io. Does anybody have any idea why it fares so poorly in Google results? Is this just due to the fact that it's a fairly new site, with few external sites linking to it? How can we improve the situation?

I hate the idea of investing large amounts of energy into writing typing docs, only for no one to see them.