Closed
Description
NumPy is considering this in numpy/numpy#14453. I suspect other projects will follow suite.
The tldr is that as long as we ship the cythonized code, our older sdists have no hope of working with newer Pythons. The have to be compiled with a modern enough Cython.
I think this makes even more sense for us. We already require that NumPy is present as a build dependency. Requiring Cython on top of that is not a big deal.
Do we want to do this for 0.25.2?
Metadata
Metadata
Assignees
Type
Projects
Relationships
Development
No branches or pull requests
Participants
Activity
jreback commentedon Sep 8, 2019
+1 on this; sure could do this for 0.25.2
WillAyd commentedon Sep 8, 2019
Makes sense to me. I suppose has the advantage of making these distributions smaller too, right?
jbrockmendel commentedon Sep 9, 2019
I don't think this is a problem, but it's worth mentioning that we tend to bump cython min-version relatively frequently in part because its not a requirement.
TomAugspurger commentedon Sep 9, 2019
I also don't think that's a problem. With pip's build isolation, a
pip install --no-binary=pandas pandasshould pull down a completely independent environment. So you should still be able to have an older Cython in your application / library's dependencies even when you're building pandas from source.BLD: Stop shipping cythonized files in sdist
jorisvandenbossche commentedon Sep 11, 2019
I am fine with no longer shipping the cythonized files, but wondering if it is a good idea to do this in a bug fix release ..
I suppose the reason to actually do it now already for 0.25.2 is because this might help for python 3.8, and 0.25.2 will probably be the first release to support this?
TomAugspurger commentedon Sep 11, 2019
I share that concern, and yes.
So if we don't want to backport this, we would just need to ensure that the environment used to build the sdist has a new enough Cython (which I've messed up in the past). Overall, I think it's best to remove, even in the bugfix release.
jorisvandenbossche commentedon Sep 11, 2019
Could we also remove the files and add cython to setup_requires, and backport this, and keep the pyproject.toml for 1.0 ? (as I think it is mainly this that can give troubles) Or is that only making a bigger mess?
TomAugspurger commentedon Sep 11, 2019
Do you know what setuptools / pip does if the user has a
requirements.txtrequiring a version of Cython that conflicts with what we would put in oursetup_requires? https://setuptools.readthedocs.io/en/latest/setuptools.html saysI'm trying it locally, and thing seem ok using
setup_requires. This is all confusing though.TomAugspurger commentedon Sep 11, 2019
I confused myself.
setup_requiresdoes not work.With these changes
I build an sdist with
python setup.py sdist. I verify that the compiled files are not present.And install
The relevant parts
So Cython apparently isn't installed (or if it is, it's not available for import).
TomAugspurger commentedon Sep 11, 2019
And if we can't import it, we can't cythonize it. I think we would need to add Cython to
install_requiresas well, but I really don't think we want to do that (especially when pyproject.toml is supposed to fix this).So tldr: I don't think setup_requires is a solution. I think our options are
min_cython_versionpip install pandasstill works.rgommers commentedon Sep 11, 2019
That sounds about right to me. Without
pyproject.tomlyou'll need to usesetup_requires(no need forinstall_requiresI think), but that invokeseasy_installbecausepipdoesn't supportsetup_requiresnatively. Which would not be good in 2019....pyproject.tomldoes have the potential to create problems, a little risky in a bugfix release. you could also go with just being careful for 0.25.2, and if another cython fix >0.29.13 is needed then do a 0.25.3jorisvandenbossche commentedon Sep 11, 2019
Yes, that's what we decided in the dev call we just had, I think. For 0.25.2 do what we have done in the past (ship cythonized files), and then on master (for 1.0) add pyproject.toml again, which also enables to properly tackle this issue of needing cython in the build step.
BLD: Add pyproject.toml (pandas-dev#28374)
BLD: Add pyproject.toml (pandas-dev#28374)
TomAugspurger commentedon Jan 10, 2020
For reference, this was done on the 1.0.0rc with pandas-dev/pandas-release@62a2a72
BLD: Add pyproject.toml (#28374)