pip build command

[pradyunsg]

Phase 2 from #5407

Allow pip to build packages for distribution - pip build

Some notes in #5407 (comment)

---

pip now has PEP 517 support and uses build isolation by default on packages with pyproject.toml files -- which means the way pip builds is difficult to reproduce externally in a trivial manner.

The pip build command would use the same code paths as PEP 517 with the sole difference being that it does not install the packages. It's essentially the pip wheel in a PEP 517 world.

[pradyunsg]

Related Discussion: pypa/packaging-problems#219

[pfmoore]

Some things that need to be considered, transferred from that discussion:

1. Presumably pip build should build wheels and sdists - whether both formats or just one to be controlled by the user.
2. This introduces a huge overlap with pip wheel, so do we remove/deprecate pip wheel?
3. However, pip wheel will download wheels rather than building them if possible - this seems to go against the intent of a pip build command, so do we drop/relocate the functionality of "download existing wheels"? There's also pip download --only-binary :all:.
4. Should pip build build dependencies as well? pip wheel currently does, as a precedent. But setup.py sdist doesn't, as a precedent for the alternative behaviour...
5. We need to make sure we don't lose the ability to support the use case of "get wheels for the following requirements (and their dependencies) by the quickest method possible, only building if necessary", as that's a core use case for populating a local wheelhouse for offline use.

Overall, I think pip needs a review of all of its download/build functionality, with the intention of rationalising it into a better organised set of subcommands.

[dholth]

I've never been convinced that building the sdist was analogous to building the binary distribution or building to install "setup.py install". It would make sense for one command to be able to create more than one kind of binary.

[pradyunsg]

Get wheels for the following requirements (and their dependencies) by the quickest method possible, only building if necessary

Do you reckon the following is acceptable as an alternative to the current pip wheel way of doing this?

$ pip download -r requirements.txt
$ pip build *.tar.gz *.zip

[pfmoore]

I hadn't really thought through how it would get done, but yes, that seems simple enough to be an acceptable approach.

Worth documenting as part of the transition, though - it's not immediately obvious that the average user would think of this. (More accurately I didn't think of it and I'd prefer to believe against all the odds that this means that it's not obvious, rather than just that I'm dumber than average 😉)

[pradyunsg]

I think the more interesting bit is that we now have 2 kinds of dependencies - build time and run time. This means even a pip build that does just "build a wheel from a source distribution" would still need all of our PackageFinder related options (AKA "index options"). :)

I have a draft structure in my head -- I'll post it when I find a bit of time to flesh it out.

[pradyunsg]

FTR - I've forgotten what I had in mind but I'll be working on pip's build logic in the coming months so I might pick this up.

[pganssle]

I would like to suggest that this is probably a bad idea at least if the motivation is to create a recommended build tool to replace setup.py sdist bdist_wheel. I think instead we should have a simple dedicated tool for this purpose rather than a pip subcommand.

One reason for this is that build doesn't obviously fit in either pip or twine - the only reason it makes sense for pip to have a build command is that pip already needs the ability to build packages in order to install them from source, but pip is generally a tool for consumers of packages, not for producers of packages. twine is usually for producers of packages but it has historically only acted upon already existing distributions and so while conceptually it makes sense for twine to be able to do the builds, it would in some sense change the nature of the tool from a tool that uploads things to PyPI into one that builds packages and uploads them to PyPI. To me, the fact that build has a reasonable claim to live in either package makes me think that build is its own thing and should be given its own package.

I'll note that while it seems like pip "already needs the capability to do builds", it in fact currently doesn't need the ability to build source distributions, since it is perfectly capable of building wheels directly from the source. tox both needs and has the ability to build source distributions, but I would not really suggest adding a tox build command to expose this functionality.

Another reason to avoid adding this to pip is that pip already has some unifying themes to its subcommands, including huge reams of command line arguments, some shared and some not. For example, the pip wheel command already exists and is quite valuable for marshalling all the build artifacts one needs to install a given package because it gets you a wheel for the thing you want to install and all its dependencies; this is, however, a terrible and counter-intuitive behavior for a tool for building a wheel for upload - you need to do the magical pip wheel --no-deps -w dist/ . invocation to do what most people actually want to do. If we add a pip build command, we'll suddenly either have to deprecate the existing and useful pip wheel command or we'll confuse everyone by having two slightly different commands with very different purposes and semantics.

The one time where I think it would make sense for pip to have a pip build command would be if pip wants to move to a situation where instead of source builds going source → wheel → install, they would go source → sdist → wheel → install (for a consistent flow that produces artifacts at each stage), that would make sense, but even then I wouldn't want to recommend it as "the way to create distributable artifacts", but rather as a tool for marshalling resources.

[merwok]

It would made sense to me if twine grew or used a PEP 517 builder to produce wheels. In some simple projects where I use flit, the same tool builds and uploads dists.