BUG: scipy-1.14.1-cp312-cp312-macosx_12_0_arm64 wheel rejected by pip check

[duncanmmacleod]

Describe your issue.

The wheel installed by pip install scipy==1.14.1 on an arm64 macos machine running macos 12.7.5 is rejected by pip check because the Tag metadata isn't recognised:

$ uname -a
Darwin builder-osx-7.ligo.org 21.6.0 Darwin Kernel Version 21.6.0: Wed Apr 24 06:05:12 PDT 2024; root:xnu-8020.240.18.708.4~1/RELEASE_ARM64_T8101 arm64
$ sw_vers
ProductName:    macOS
ProductVersion: 12.7.5
BuildVersion:   21H1222
$ conda create -n test -c conda-forge python=3.12 pip
$ conda activate test
$ pip install scipy
$ pip list
Package    Version
---------- -------
numpy      2.1.0
pip        24.2
scipy      1.14.1
setuptools 72.2.0
wheel      0.44.0
$ pip check
scipy 1.14.1 is not supported on this platform

The issue, as I understand it, is that the Tag metadata for the wheel doesn't match any of the tags understood by the packaging library (vendored inside pip):

$ grep Tag ${CONDA_PREFIX}/lib/python3.12/site-packages/scipy-*.dist-info/WHEEL
Tag: cp312-cp312-macosx_12_3_arm64
$ python3 -c "import pprint; from pip._vendor.packaging.tags import mac_platforms; pprint.pprint(list(mac_platforms()))";
['macosx_12_0_arm64',
 'macosx_12_0_universal2',
 'macosx_11_0_arm64',
 'macosx_11_0_universal2',
 'macosx_10_16_universal2',
 'macosx_10_15_universal2',
 'macosx_10_14_universal2',
 'macosx_10_13_universal2',
 'macosx_10_12_universal2',
 'macosx_10_11_universal2',
 'macosx_10_10_universal2',
 'macosx_10_9_universal2',
 'macosx_10_8_universal2',
 'macosx_10_7_universal2',
 'macosx_10_6_universal2',
 'macosx_10_5_universal2',
 'macosx_10_4_universal2']

i.e. there is no valid macosx_12_3_arm64 platform.

I have no idea whether this is a bug in pip/packaging or an issue with that wheel.

Reproducing Code Example

conda create -n test -c conda-forge python=3.12 pip
conda activate test
pip install scipy
pip check

Error message

scipy 1.14.1 is not supported on this platform

SciPy/NumPy/Python version and system information

1.14.1 2.1.0 sys.version_info(major=3, minor=12, micro=5, releaselevel='final', serial=0)
/Users/duncanmacleod/opt/conda/envs/test/lib/python3.12/site-packages/scipy/__config__.py:154: UserWarning: Install `pyyaml` for better output
  warnings.warn("Install `pyyaml` for better output", stacklevel=1)
{
  "Compilers": {
    "c": {
      "name": "clang",
      "linker": "ld64",
      "version": "15.0.0",
      "commands": "cc"
    },
    "cython": {
      "name": "cython",
      "linker": "cython",
      "version": "3.0.11",
      "commands": "cython"
    },
    "c++": {
      "name": "clang",
      "linker": "ld64",
      "version": "15.0.0",
      "commands": "c++"
    },
    "fortran": {
      "name": "gcc",
      "linker": "ld64",
      "version": "12.1.0",
      "commands": "gfortran"
    },
    "pythran": {
      "version": "0.16.1",
      "include directory": "../../../../../../private/var/folders/hw/1f0gcr8d6kn9ms0_wn0_57qc0000gn/T/pip-build-env-x9rig1ei/overlay/lib/python3.12/site-packages/pythran"
    }
  },
  "Machine Information": {
    "host": {
      "cpu": "aarch64",
      "family": "aarch64",
      "endian": "little",
      "system": "darwin"
    },
    "build": {
      "cpu": "aarch64",
      "family": "aarch64",
      "endian": "little",
      "system": "darwin"
    },
    "cross-compiled": false
  },
  "Build Dependencies": {
    "blas": {
      "name": "scipy-openblas",
      "found": true,
      "version": "0.3.27.dev",
      "detection method": "pkgconfig",
      "include directory": "/private/var/folders/hw/1f0gcr8d6kn9ms0_wn0_57qc0000gn/T/cibw-run-ghg42m3n/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/scipy_openblas32/include",
      "lib directory": "/private/var/folders/hw/1f0gcr8d6kn9ms0_wn0_57qc0000gn/T/cibw-run-ghg42m3n/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/scipy_openblas32/lib",
      "openblas configuration": "OpenBLAS 0.3.27.dev DYNAMIC_ARCH NO_AFFINITY neoversen1 MAX_THREADS=64",
      "pc file directory": "/Users/runner/work/scipy/scipy"
    },
    "lapack": {
      "name": "scipy-openblas",
      "found": true,
      "version": "0.3.27.dev",
      "detection method": "pkgconfig",
      "include directory": "/private/var/folders/hw/1f0gcr8d6kn9ms0_wn0_57qc0000gn/T/cibw-run-ghg42m3n/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/scipy_openblas32/include",
      "lib directory": "/private/var/folders/hw/1f0gcr8d6kn9ms0_wn0_57qc0000gn/T/cibw-run-ghg42m3n/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/scipy_openblas32/lib",
      "openblas configuration": "OpenBLAS 0.3.27.dev DYNAMIC_ARCH NO_AFFINITY neoversen1 MAX_THREADS=64",
      "pc file directory": "/Users/runner/work/scipy/scipy"
    },
    "pybind11": {
      "name": "pybind11",
      "version": "2.12.0",
      "detection method": "config-tool",
      "include directory": "unknown"
    }
  },
  "Python Information": {
    "path": "/private/var/folders/hw/1f0gcr8d6kn9ms0_wn0_57qc0000gn/T/cibw-run-ghg42m3n/cp312-macosx_arm64/build/venv/bin/python",
    "version": "3.12"
  }
}

[dschmitz89]

Cc @tylerjereddy

[andyfaff]

Yes, I can see that. I think we might have to go into the wheel and adjust the tag somehow.

[andyfaff]

The background to this is that github actions only has M-series build images for macos-14. When we build on that image we have to set MACOS_DEPLOYMENT_TARGET=12.3 because we can't actually get the tools to build as old as 12.0, and delocate throws an error if the dylibs are built for 12.3, but we try to make a 12_0 wheel name.

This then means that the wheel has 12_3 in it's name, which is invalid as well. We therefore rename the wheel to 12_0. It seems that the tag inside the wheel, in the WHEEL file, also needs amending. We might be able to wheel unpack, amend the WHEEL file, then repack. That's a whole lot of pain, it would be easier if there was a utility to do that. I tried using wheel tags --platform-tags, but I kept getting a hash mismatch message, so some hash might need to be adjusted somewhere.

However, having said that I believe that the wheel is still probably usable, even though it fails pip check. I tried installing a scipy-1.14.1-cp311-cp311-macosx_12_0_arm64.whl onto a macOS-14 computer locally. The wheel installs, fails pip check, but passes the test suite.

[andyfaff]

(It'd be easier if we made a minimum target of 13_0 for macos_arm64).

[duncanmmacleod]

@andyfaff, should packaging be updated to support macosx_12_3_arm64 as a platform? It is currently hardcoded to only recognise 0 as the minor version.

https://github.com/pypa/packaging/blob/ead9d34f9b6a5e504a4563d9246f76d37a112e85/src/packaging/tags.py#L444-L452

[andyfaff]

I certainly don't think we're building in the nicest of ways, but I don't think we could've done much else apart from make macos 13 the minimum version.

Im not sure that adding an extra tag to packaging solves things, I don't know enough about that.

[tylerjereddy]

I don't have much to add at the moment. If the installed wheel is viable apart from failing pip check it doesn't seem like an emergency, but perhaps makes sense to keep this open to see if we can do better in the future.

[rgommers]

Agreed, not an emergency, just a thing that would be nice to solve for the next release.

I'm not sure that adding an extra tag to packaging solves things,

We can't easily go add the missing tags there; only supporting major macOS releases is the current design choice. Improving that to take minor versions into account is discussed at pypa/packaging#578 (content warning: long and complex change).

(It'd be easier if we made a minimum target of 13_0 for macos_arm64).

I think that that would be fine probably. It'd be nicer if that wasn't necessary though. That's perhaps possible:

• xref BLD: make cp313 wheels [wheel build] #21000 (comment) for why this is a problem to begin with (libquadmath deployment target is 12.3)
• xref BLD: use OpenBLAS v0.3.28 with fewer kernels, fix OpenBLAS licences #21360 (comment) for getting rid of libquadmath