Support for pip install --build-options / --global-options / --install-options

[pmav99]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Question

There are some packages that in order to be installed you need to pass additional options to setup.py. There is support for this in pip. E.g. to install GDAL on Ubuntu 18.04 you need to install gdal-bin and libgdal-dev using apt and then to issue:

pip install --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL==`gdal-config --version`

The question is, does poetry support this? Is there a way to specify these options in pyproject.toml? If not, how do you suggest handling this case in a project that depends on such packages?

[pakal]

I confirm that without such options, some packages like Pillow, rcssmin etc. can be a pain to install, especially when they try to build optional C extensions on environments where it's not possible. Having these pip-like options to be put in the listing of dependencies would be a great feature.

[higherorderfunctor]

I would also like to see this supported in native poetry projects so the options are passed to the undocumented build property for building C extensions.

Ideally I could have one poetry project that builds a C extension. Then in another poetry project, pass the build options.

[tool.poetry.dependencies]
python = "^3.7"
other-project = { git = "ssh://.../other-project.git", branch = "master", install-option="--some-lib-prefix=/opt/some-lib" }

[pmav99]

Could someone provide some hint on how to tackle this? Where would we need to make changes in order to allow for the syntax proposed by @higherorderfunctor ?

[higherorderfunctor]

I did some preliminary exploring. Assuming each dependency is installed via a different call to PIP, then those args would need to be passed at this line.
https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/pip_installer.py#L115

There are two paths that define *args that can be found here and here:
https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/pip_installer.py#L41
https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/pip_installer.py#L182

The entry point for these paths is here and here. I am assuming those options would need to be extracted from the package argument and included in *args:
https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/pip_installer.py#L30
https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/pip_installer.py#L170

The toml schema for dependencies is here which I assume needs to be aware of the two options:
https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/json/schemas/poetry-schema.json#L193

The dependency definition that I assume is passed to the install commands above is located here which would need to be aware of the options:
https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/packages/dependency.py#L21

Dependencies are built here which looks to parse the options out of the toml file and build the dependency definition which would need to be updated:
https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/packages/package.py#L264

I haven't fully traced the life-cycle of a Dependency to the PipInstaller. It looks like Dependencies are stored on the package here (I don't think any code change would be needed here):
https://github.com/sdispater/poetry/blob/master/poetry/packages/package.py#L68

Dependencies look to be "converted" to packages here (no code change needed I can see):
https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/installer.py#L495

More maybe needed... such as ensuring those options are exposed post-build.

[jkyl]

@higherorderfunctor thanks for the exploration, that is very helpful. I am interested in something quite similar: passing a link to pip install --find-links. This would enable me to pull the appropriate .whl for pytorch on Windows (pytorch's pypi only contains linux/mac .whls).

[jkyl]

#1511

[reedjosh]

#2017

[Matesanz]

Thanks for the Feature!
PR says documentation on this was updated but I'm not able to find it, could you please provide the link or an example to this feature? Thanks in advance.

[reedjosh]

Hey, sorry. I think I just left the tic-boxes checked when I submitted the template #2017

I did not add tests or documentation. I would be happy to add these items, but I wanted to verify that the PR was using acceptable methods before moving forward.

I will continue the discussion there.

[davidbeers]

Lack of support for these pip options is the main thing that holds me back from adopting Poetry right now. Like @pmav99 I need the gdal package and it can't be installed properly without the mentioned options.