Pedantic `IncompatibleConstraintsError` with `scipy` 1.11.2 on Debian

[isosphere]

• Poetry version: 1.6.1

• Python version: 3.10.13 (installed from source)

• OS version and name: Debian 12.1 (bookworm), x86_64

• pyproject.toml: https://raw.githubusercontent.com/isosphere/VolAdjGIPDashboard/master/pyproject.toml

• I am on the latest stable Poetry version, installed using a recommended method.

• I have searched the issues of this repo and believe that this is not a duplicate.

• I have consulted the FAQ and blog for any relevant entries or release notes.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

poetry env info                                                                                                                                              09/05/2023 01:39:48 PM

Virtualenv
Python:         3.10.13
Implementation: CPython
Path:           /var/vegiportfolio/VolAdjGIPDashboard/.venv
Executable:     /var/vegiportfolio/VolAdjGIPDashboard/.venv/bin/python
Valid:          True

System
Platform:   linux
OS:         posix
Python:     3.10.13
Path:       /usr/bin
Executable: /usr/bin/bin/python3.10

pip 23.2.1

 poetry install -vvv                                                                                                                                          09/05/2023 01:38:22 PM
Loading configuration file /home/matt/.config/pypoetry/config.toml
Using virtualenv: /var/vegiportfolio/VolAdjGIPDashboard/.venv
Installing dependencies from lock file
Warning: poetry.lock is not consistent with pyproject.toml. You may be getting improper dependencies. Run `poetry lock [--no-update]` to fix it.

Finding the necessary packages for the current system

Package operations: 12 installs, 0 updates, 0 removals, 65 skipped

  • Installing scipy (1.11.2): Pending...
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
No suitable keyring backend found
No suitable keyring backends were found
Keyring is not available, credentials will be stored and retrieved from configuration files as plaintext.
Creating new session for pypi.org
Skipping wheel scipy-1.11.2-cp310-cp310-macosx_10_9_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp310-cp310-macosx_12_0_arm64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp310-cp310-musllinux_1_1_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp310-cp310-win_amd64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp311-cp311-macosx_10_9_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp311-cp311-macosx_12_0_arm64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp311-cp311-musllinux_1_1_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp311-cp311-win_amd64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp312-cp312-macosx_10_9_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp312-cp312-macosx_12_0_arm64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp312-cp312-musllinux_1_1_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp312-cp312-win_amd64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp39-cp39-macosx_10_9_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp39-cp39-macosx_12_0_arm64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp39-cp39-musllinux_1_1_x86_64.whl as this is not supported by the current environment
Skipping wheel scipy-1.11.2-cp39-cp39-win_amd64.whl as this is not supported by the current environment
  • Installing scipy (1.11.2): Preparing...
  • Installing scipy (1.11.2): Failed

  Stack trace:

  18  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:286 in _execute_operation
       284│
       285│             try:
     → 286│                 result = self._do_execute_operation(operation)
       287│             except EnvCommandError as e:
       288│                 if e.e.returncode == -2:

  17  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:386 in _do_execute_operation
       384│             return 0
       385│
     → 386│         result: int = getattr(self, f"_execute_{method}")(operation)
       387│
       388│         if result != 0:

  16  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:511 in _execute_install
       509│
       510│     def _execute_install(self, operation: Install | Update) -> int:
     → 511│         status_code = self._install(operation)
       512│
       513│         self._save_url_reference(operation)

  15  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:549 in _install
       547│             archive = self._download_link(operation, Link(package.source_url))
       548│         else:
     → 549│             archive = self._download(operation)
       550│
       551│         operation_message = self.get_operation_message(operation)

  14  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:753 in _download
       751│             self._yanked_warnings.append(message)
       752│
     → 753│         return self._download_link(operation, link)
       754│
       755│     def _download_link(self, operation: Install | Update, link: Link) -> Path:

  13  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:800 in _download_link
       798│             self._write(operation, message)
       799│
     → 800│             archive = self._chef.prepare(archive, output_dir=output_dir)
       801│
       802│         # Use the original archive to provide the correct hash.

  12  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chef.py:101 in prepare
        99│             return self._prepare(archive, destination=destination, editable=editable)
       100│
     → 101│         return self._prepare_sdist(archive, destination=output_dir)
       102│
       103│     def _prepare(

  11  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chef.py:185 in _prepare_sdist
       183│             destination.mkdir(parents=True, exist_ok=True)
       184│
     → 185│             return self._prepare(
       186│                 sdist_dir,
       187│                 destination,

  10  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chef.py:116 in _prepare
       114│                 runner=quiet_subprocess_runner,
       115│             )
     → 116│             env.install(builder.build_system_requires)
       117│
       118│             stdout = StringIO()

   9  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chef.py:80 in install
        78│         )
        79│         installer.update(True)
     →  80│         installer.run()
        81│
        82│

   8  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/installer.py:104 in run
       102│             self.verbose(True)
       103│
     → 104│         return self._do_install()
       105│
       106│     def dry_run(self, dry_run: bool = True) -> Installer:

   7  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/installer.py:241 in _do_install
       239│                 source_root=self._env.path.joinpath("src")
       240│             ):
     → 241│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       242│         else:
       243│             self._io.write_line("Installing dependencies from lock file")

   6  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/puzzle/solver.py:72 in solve
        70│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        71│             start = time.time()
     →  72│             packages, depths = self._solve()
        73│             end = time.time()
        74│

   5  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/puzzle/solver.py:155 in _solve
       153│
       154│         try:
     → 155│             result = resolve_version(self._package, self._provider)
       156│
       157│             packages = result.packages

   4  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/mixology/__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│
     →  18│     return solver.solve()
        19│

   3  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/mixology/version_solver.py:164 in solve
       162│             while next is not None:
       163│                 self._propagate(next)
     → 164│                 next = self._choose_package_version()
       165│
       166│             return self._result()

   2  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/mixology/version_solver.py:503 in _choose_package_version
       501│             package = locked
       502│
     → 503│         package = self._provider.complete_package(package)
       504│
       505│         conflict = False

   1  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/puzzle/provider.py:609 in complete_package
       607│             # For dependency resolution, markers of duplicate dependencies must be
       608│             # mutually exclusive.
     → 609│             deps = self._resolve_overlapping_markers(package, deps)
       610│
       611│             if len(deps) == 1:

  IncompatibleConstraintsError

  Incompatible constraints in requirements of -root- (0.0.0):
  numpy (==1.22.0) ; platform_machine == "loongarch64"
  numpy (==1.22.3) ; python_version == "3.10" and platform_system == "Windows" and platform_python_implementation != "PyPy"

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/puzzle/provider.py:913 in _resolve_overlapping_markers
      909│                     specific_source_dependency = dep
      910│                 constraint = constraint.intersect(dep.constraint)
      911│             if constraint.is_empty():
      912│                 # conflict in overlapping area
    → 913│                 raise IncompatibleConstraintsError(package, *used_dependencies)
      914│
      915│             if not any(uses):
      916│                 # This is an edge case where the dependency is not required
      917│                 # for the resulting marker. However, we have to consider it anyway

This is an x86_64 system on Debian Linux - the above restrictions are irrelevant, or at least the meaning of this error is misleading.

numpy 1.25.2 is installed in the virtual env.

I've reinstalled poetry, and also the virtual env, to no avail. Not sure how to move forward. I've tried specifying different scipy versions, but I just get different variations on the above. I don't understand how constraints that don't apply to my system are preventing installation.

Any advice appreciated!

Possibly related to #7257.

[dimbleby]

poetry builds cross-platform solutions - even for build environments - and has detected and reported a possible combination of markers with conflicting requirements.

looks like the scipy build requirements are just bugged, these lines should include something about platform_machine!='loongarch64' (just as the set that follows does)

possibly "python installed from source" is contributing to making this hard for yourself, you likely ought to be able to use one of the scipy binary distributions and I can only guess that there's something about your build that prevents this.

[isosphere]

Looks like scipy agrees with your assessment!

Poetry is technically right, but it's also being pedantic. I imagine there's a good reason for building architecture-agnostic solutions, but I'm just trying to manage packages for deployment on the architectures I'm using, which is x86_64.

Is there any possibility of dismissing this kind of error from Poetry, perhaps by specifying in pyproject.toml that this project is restricted to specific architectures, so it doesn't need to bug me about it being impossible to deploy on others?

[radoering]

poetry builds cross-platform solutions - even for build environments

Actually, I think it's not necessary to solve the build requirements cross-platform because we only build on demand during installation for the current platform (after creating the lockfile for the project). We just did it because it was simpler to implement. At the moment locking is done implicitly here:

poetry/src/poetry/installation/chef.py

Lines 70 to 80 in 5986df0

	installer = Installer(
	NullIO(),
	self._env,
	package,
	Locker(self._env.path.joinpath("poetry.lock"), {}),
	self._pool,
	Config.create(),
	InstalledRepository.load(self._env),
	)
	installer.update(True)
	installer.run()

... and we have no parameters to say "lock only for the current platform". I think in case of the build requirements it would make sense since the lockfile is only temporary so there is no danger that someone picks up the lockfile and uses it on another platform. But as always: the issue had to become important enough for someone who's capable to solve it.

[dadudadn]

I have a similar problem with scipy. the error message in the log is entirely correct ==> but it seems to point to the code not resolving the backwards range that scipy has posted on pypi. Most post their compatibility range starting low(>=3.8) and ending high(< 3.12). I suspect poetry reads it wrong.

here's part of the log:

" SolveFailure

The current project's Python requirement (>=3.10) is not compatible with some of the required packages Python requirement:
- scipy requires Python <3.12,>=3.8, so it will not be satisfied for Python >=3.12

Because pkg depends on scipy (1.10.1) which requires Python <3.12,>=3.8, version solving failed.

at ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/mixology/version_solver.py:416 in _resolve_conflict"

[BeRT2me]

@dadudadn

$ poetry add scipy@latest
Using version ^1.11.3 for scipy

Updating dependencies
Resolving dependencies... (0.0s)

The current project's Python requirement (>=3.10,<4.0) is not compatible with some of the required packages Python requirement:
  - scipy requires Python <3.13,>=3.9, so it will not be satisfied for Python >=3.13,<4.0

Because no versions of scipy match >1.11.3,<2.0.0
 and scipy (1.11.3) requires Python <3.13,>=3.9, scipy is forbidden.
So, because ai-testing depends on scipy (^1.11.3), version solving failed.

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties
    
    For scipy, a possible solution would be to set the `python` property to ">=3.10,<3.13"

    https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies,
    https://python-poetry.org/docs/dependency-specification/#using-environment-markers

Poetry states to specify the python requirement via the python or markers properties. Suggesting, for scipy, to set its python property to >=3.10,<3.13.

It then provides links to how to do this using either "Python restricted dependencies" or "environment markers".

The very next entry in their documentation is for Multiple constraints dependencies, with which you can add scipy as a * dependency for >=3.13 if you so please. There really isn't any other alternative since the version that will support 3.13 is unknown.

---

This is likely what you want.

[tool.poetry.dependencies]
python = ">=3.10"
scipy = {version = "^1.10.1", python = ">=3.10,<3.12"}

Or, if you want to be fancy...

[tool.poetry.dependencies]
python = "^3.10"
scipy = [
    {version = "^1.10.1", python = ">=3.10,<3.12"},
    {version = "^1.11.3", python = "3.12"},
    {version = "*", python = ">=3.13"},
]

[dadudadn]

@BeRT2me did you continue your test? Did you try to install? I'd already done both for python and scipy as you've outlined in the toml file. I'm talking about a bug in Poetry code-- and plainly stated in poetry's own error log -- If you read it slowly and closely -- this is the error: "

The current project's Python requirement (>=3.10) is not compatible with some of the required packages Python requirement:
scipy requires Python <3.12,>=3.8, so it will not be satisfied for Python >=3.12

Because pkg depends on scipy (1.10.1) which requires Python <3.12,>=3.8, version solving failed.

Notice how scipy's Python requirement is NOT the usual order(start with the lower limit, end with the upper limit) and poetry fails to recognize that difference in it's message? Poetry looks at the first limit as the "lower" limit, so it fails to add scipy, even after I've specifed everything correctly in the toml for both python and scipy
BUG

Numpy gives me the same problem, on "poetry add numpy" but it will install it as a requirement for another package.. And numpy has the same backwards python-required-range on pypi.org.

[dimbleby]

@dadudadn what you are reporting is both (i) not what this issue describes - this is specifically about build requirements - and (ii) not a bug.

Please read the FAQ and stop hijacking this issue.

[BeRT2me]

@dadudadn both of my configs work perfectly fine.

You say python >= 3.10,<4 (^3.10). This means you say your project works with python >= 3.13. If your project requires scipy, it does not work with python >= 3.13, therefore, poetry will not add it. Same goes for NumPy. That is not a bug.