RLS: 2.3.1

[mroeschke]

Placeholder issue if we decide to release 2.3.1. At the time of writing this issue, it's expected that pandas 3.0 would be the next version #57064

Notable tasks for 2.3.1:

• Re-enable Python 3.9 support (Failed to install pandas==2.3.0 with Python 3.9 #61563, pandas.__version__ is 2.3.0+4.g1dfc98e16a in pandas 2.3.0 and python 3.9, not 2.3.0 #61579)
  • Revert [2.3.x] DEPS: Drop Python 3.9 #60792
  • Merge BLD: Build wheels for 3.9 and musllinux-aarch64 for pandas 2.3 #61569 (without hardcoding version)
• Re-enable musl-aarch64 wheels
  • Merge BLD: Build wheels for 3.9 and musllinux-aarch64 for pandas 2.3 #61569 (without hardcoding version)

[simonjayhawkins]

Thanks @mroeschke for opening this issue.

Placeholder issue if we decide to release 2.3.1.

RLS 2.3 was driven by the need to release to the community for evaluation/experimentation the new np.nan variant of the nullable string dtype (both object fallback and pyarrow backed).

So it would be, to me, a reasonable assumption that any issues reported regarding this new dtype would be fixed in the 2.3.x branch. Not providing these fixes to the community until the 3.0 release candidate would not allow time for proper evaluation/feedback?

The new string dtype is behind a feature flag and experimental so changes to behavior could be added in a patch release? and not need to wait for a minor/major release?

PDEP-14 states "The 2.3.0 release would then have all future string functionality available". Did all the issues related to the new dtype get into 2.3.0? I can see there are a few stale PRs for the string dtype that presumably still need to be resolved. These could be included in a patch release?

So at this time i think it may be appropriate to assume that we will need to actively maintain the 2.3.x branch and that patch releases are more likely than not?

At the time of writing this issue, it's expected that pandas 3.0 would be the next version #57064

releasing 3.0 before an reasonable amount of time has elapsed would defeat the whole purpose and exercise of creating a 2.3 release. Some contributors have suggested a release cadence of 4 months between minor releases, but 6 months is probably more realistic?

There is a deprecation in 2.3 related to str.contains. It may not be relevant or too significant but our deprecation policy, PDEP-17 states:

Deprecated functionality should remain unchanged in at least 2 minor releases before being changed or removed
Deprecations should initially use DeprecationWarning, and then be switched to FutureWarning in the last minor release before the major release they are planned to be removed in

#59615 used a FutureWarning so we don't need to consider the 2nd clause. However, to honor the first clause may strictly require a 2.4 release before the behavior is then changed in 3.0? A pragmatic approach may be needed but also consideration to not making a mockery of the deprecation policy. Or just postpone enforcing that deprecation until 4.0.

#59328 gives an overview of breaking changes

[simonjayhawkins]

#59328 gives an overview of breaking changes

There was consideration to including the breaking changes in the 2.3 release notes. This wasn't done in time for the 2.3 an hence the changes not yet communicated to the community.

[simonjayhawkins]

@mroeschke ive installed 2.3 and then installed pyarrow using mamba. The pyarrow version installed is 20.0.0.

It seems this combo doesn't work? Is there any fixes that should be backported or changes to conda-forge instead?

[mroeschke]

It seems this combo doesn't work? Is there any fixes that should be backported or changes to conda-forge instead?

What error shows up they are installed together? There's a possibly the conda-forge recipe is kinda out of sync with our dependencies in pyproject.toml

[simonjayhawkins]

so i've just switched kernels back after using pyArrow 19.0.1 and the notebook is running fine now with 20.0.0. So it looks like it was an issue on my end.

[jorisvandenbossche]

On the technical side, I think I have backported all PRs that we forgot to backport for 2.3.0 or that were merged since 2.3.0 to main and should target 2.3.1 (based on our "Still Needs Manual Backport" label).
I think that mostly leaves the CI / packaging (3.9 wheels) related changes that we should include, listed in the top post (@mroeschke do you have some time to look at those?)

On the communication side, I would still like to update the 2.3.0 release notes to include more details about the upcoming changes, because right now we didn't actually really communicate about this when releasing 2.3.0. I am currently first creating this content for the 3.0.0 release notes (#61724), and when that is done I would also add a distilled version of that to the 2.3.0 and 2.3.1 release notes, mentioning the opt-in feature flags.
And that content also refers to the migration guide I started at #61705. Feedback and proofreading of those documentation proposals is very welcome.

Regarding a timeline for 2.3.1, I think we already have enough changes to warrant a release (once the packaging/CI issues are resolved), and personally I would have time for this first half of next week.

[simonjayhawkins]

Regarding a timeline for 2.3.1, I think we already have enough changes to warrant a release (once the packaging/CI issues are resolved), and personally I would have time for this first half of next week.

+1

[simonjayhawkins]

On the communication side, I would still like to update the 2.3.0 release notes to include more details about the upcoming changes, because right now we didn't actually really communicate about this when releasing 2.3.0. I am currently first creating this content for the 3.0.0 release notes (#61724), and when that is done I would also add a distilled version of that to the 2.3.0 and 2.3.1 release notes, mentioning the opt-in feature flags.
And that content also refers to the migration guide I started at #61705. Feedback and proofreading of those documentation proposals is very welcome.

@jorisvandenbossche it would be good to get that into 2.3.1 but maybe not a blocker. Hopefully I will be able to help and to go over your docs if not this week, maybe at the weekend. But yes, if these can be included in 2.3.1 also, that'll be a bonus.