Add support for outputting a list of extras and their requirements.

[software-opal]

• Pip version: 9.0.1
• Python version: 3.6
• Operating system: Ubuntu 17.10

Description:

I'm trying to write a script to automatically generate constraints files(with hashes) and I'd like a way to allow this script to detect any extras, and generate the constraints file for those too.

I have no preference to the command; but it might be nice to put it in pip show as another section(depending on compatibility)

What I've run:

$ pip show requests
Name: requests
Version: 2.18.4
...
Requires: idna, urllib3, chardet, certifi
ExtraRequires[security]: cryptography, pyOpenSSL
ExtraRequires[socks]: PySocks

[software-opal]

An alternative would be to create a pip metadata <package> command to output the <...>.dist-info/METADATA or <...>.dist-info/metadata.json files so that they can be parsed by downstream tooling.

[pradyunsg]

This seems to be deeply related and a better alternative to #3797.

[jaraco]

An even better alternative would be to use importlib_metadata, which has an API.

>>> import importlib_metadata
>>> importlib_metadata.metadata('xonsh').get_all('Provides-Extra')
['linux', 'mac', 'proctitle', 'ptk', 'pygments', 'win']
>>> importlib_metadata.metadata('xonsh').get_all('Requires-Dist')
["distro; extra == 'linux'", "gnureadline; extra == 'mac'", "setproctitle; extra == 'proctitle'", "prompt-toolkit; extra == 'ptk'", "pygments (>=2.2); extra == 'pygments'", "win-unicode-console; extra == 'win'"]

[jaraco]

And use packaging to parse them:

>>> req = next(map(packaging.requirements.Requirement, importlib_metadata('xonsh').get_all('Requires-Dist')))
>>> req.name
'distro'
>>> req.specifier
<SpecifierSet('')>
>>> req.extras
set()
>>> req.marker
<Marker('extra == "linux"')>

[glyph]

The lack of this feature is an increasingly troublesome annoyance for Twisted. Since python packaging is allegedly Good Now, Actually™ (thanks for that, by the way!), we're starting to rely more on asking our users to install e.g. twisted[tls] or twisted[all_non_platform] in order to get specific optional features, rather than testing huge and ancient dependency matrices so everything will work correctly with default packages on whatever unfortunate end-of-lifed Linux distribution our users happen to be installing on.

However, how is the user supposed to know if it's twisted[tls] or twisted[ssl]? Where do they find out about the string all_non_platform? It would be really great if we could not only get a list of extras, but also like, include some documentation with each one so that as someone is pip install'ing, it's clear to them what extras are available and what each one means.

Thanks for reading!

[pfmoore]

Given that, as @jaraco mentioned, the data is available via importlib_metadata and packaging, maybe someone could write and publish a standalone script to extract this data?

I'm sympathetic to the idea that making the information available via pip aids discoverability, but given that it's possible to get the information the OP needed already, it seems less than ideal to block any progress on this until it can be added to pip. (If nothing else, a standalone script could be the basis of a future PR for pip).

[exarkun]

I'm sympathetic to the idea that making the information available via pip aids discoverability, but given that it's possible to get the information the OP needed already, it seems less than ideal to block any progress on this until it can be added to pip.

The OP posted the issue a year and a half ago. I hope they already found a way to not be blocked in their endeavours. :) This is the pip issue tracker though. It seems appropriate to focus on how to support this use-case in pip here.

[exarkun]

The lack of this feature is an increasingly troublesome annoyance for Twisted. Since python packaging is allegedly Good Now, Actually™ (thanks for that, by the way!), we're starting to rely more on asking our users to install e.g. twisted[tls] or twisted[all_non_platform] in order to get specific optional features, rather than testing huge and ancient dependency matrices so everything will work correctly with default packages on whatever unfortunate end-of-lifed Linux distribution our users happen to be installing on.

However, how is the user supposed to know if it's twisted[tls] or twisted[ssl]? Where do they find out about the string all_non_platform? It would be really great if we could not only get a list of extras, but also like, include some documentation with each one so that as someone is pip install'ing, it's clear to them what extras are available and what each one means.

Thanks for reading!

One thing to note is that the OP seems to have been discussing gathering this information for programmatic consumption ("I'm trying to write a script to automatically ...") but this comment from @glyph is firmly focused on user education and so probably manual usage by a human. These are both good and interesting use-cases but they may not have the same solution.

[tsvikas]

while it might be a slight different enhancement than what OP meant, I think we all would be glad to have a pip search-extras <package>, that will output the available extras for an (uninstalled) package.
pip is a tool that allows us to list (search) packages & than install them, it allows us to install extras, but how can we discover them?
I believe it's more similar to what @glyph wanted?

Using the
import importlib_metadata; importlib_metadata.metadata('twisted').get_all('Provides-Extra')
requires a venv with importlib_metadata installed, and running python code - not an easy interaction when you are just trying to remember a name of an extra to install.