-
-
Notifications
You must be signed in to change notification settings - Fork 18.8k
Description
Now we are adding ExtensionArray.argmin/argmax (#24382 / #27801), we also need to decide on its behaviour for nullable / masked arrays.
For the default of skipna=True
, I think the behaviour is clear (we skip those values, and calculate the argmin/argmax of the remaining values), but we need to decide on the behaviour in case of skipna=False
in presence of missing values.
I don't think it makes much sense to follow the current behaviour for Series with NaNs:
In [25]: pd.Series([1, 2, 3, np.nan]).argmin()
Out[25]: 0
In [26]: pd.Series([1, 2, 3, np.nan]).argmin(skipna=False)
Out[26]: -1
(the -1 is discussed separately in #33941 as well)
We also can't really compare with numpy, as there NaNs are regarded as the largest values (which follows somewhat from the sorting behaviour with NaNs).
For other reductions, the logic is: once there is an NA present, the result is also an NA with skipna=False
(with the logic here that the with NAs, the min/max is not known, and thus also not the location).
So something like:
>>> pd.Series([1, 2, 3, pd.NA], dtype="Int64").argmin(skipna=False)
<NA>
However, this also means that argmin
/argmax
return something that is not an integer, and since those methods are typically used to get a value that can be used to index, that might also be annoying.
An alternative could also be to raise an error (similar as for empty arrays).
Activity
dsaxton commentedon May 4, 2020
I think your suggestion that argmin / argmax should return NA when we have NA and skipna is False makes total sense (I would even say that any other behavior is wrong / inconsistent with how NA is already defined / implemented)
arw2019 commentedon Dec 3, 2020
I'm +1 on returning
NA
withskipna=False
cc @pandas-dev/pandas-core @pandas-dev/pandas-triage there's a PR in the works to implement this once we decide on the behavior
jbrockmendel commentedon Jul 17, 2023
Discussed on last week's dev call, agreed that argmax/argmin should always return an integer regardless of dtype. Agreed to deprecate the current cases that return -1 in favor of raising.
rhshadrach commentedon Jul 20, 2023
I'm a bit late to the party, but was idxmin/max discussed? In groupby with categorical groups and
observed=False
we need to decide on a behavior: #10694jbrockmendel commentedon Jul 20, 2023
we agreed last week that the relevant cases (where argmin/argmax return -1s) should also be deprecated and eventually raise. ive got an upcoming PR for that deprecation. Will that have downstream consquences we may not have considered?
rhshadrach commentedon Jul 20, 2023
Not necessarily - at most it would be from a high level "API Consistency" type of view, but perhaps not even that. I've never been sure what idxmin/idxmax should return in the case of a "missing value", and raising a better error message than the one that occurs in #10694 sounds like a good path forward to me.