Skip to content

BUG: MultiIndex RollingGroupby returns only one level of index #38523

@Heerozh

Description

@Heerozh
  • I have checked that this issue has not already been reported.

    I have confirmed this bug exists on the latest version of pandas.

    (optional) I have confirmed this bug exists on the master branch of pandas.


import pandas as pd
print(pd.__version__)

arrays = [['Falcon', 'Falcon', 'Parrot', 'Parrot'],
          ['Captive', 'Wild', 'Captive', 'Wild']]
index = pd.MultiIndex.from_arrays(arrays, names=('Animal', 'Type'))
df = pd.DataFrame({'Max Speed': [390., 350., 30., 20.]},
                  index=index)
print(df, '\n')
print(df.groupby(level=0)['Max Speed'].rolling(2).sum())

Problem description

Version 1.1.5 returned different result compared to earlier versions

Expected Output (0.22.0)

0.22.0
                Max Speed
Animal Type              
Falcon Captive      390.0
       Wild         350.0
Parrot Captive       30.0
       Wild          20.0 

Animal  Animal  Type   
Falcon  Falcon  Captive      NaN
                Wild       740.0
Parrot  Parrot  Captive      NaN
                Wild        50.0
Name: Max Speed, dtype: float64

Expected Output (1.1.3)

1.1.3
                Max Speed
Animal Type              
Falcon Captive      390.0
       Wild         350.0
Parrot Captive       30.0
       Wild          20.0 

Animal  Animal  Type   
Falcon  Falcon  Captive      NaN
                Wild       740.0
Parrot  Parrot  Captive      NaN
                Wild        50.0
Name: Max Speed, dtype: float64

Output (1.1.5)

1.1.5
                Max Speed
Animal Type              
Falcon Captive      390.0
       Wild         350.0
Parrot Captive       30.0
       Wild          20.0 

Animal
Falcon      NaN
Falcon    740.0
Parrot      NaN
Parrot     50.0
Name: Max Speed, dtype: float64

INSTALLED VERSIONS

commit : b5958ee
python : 3.7.9.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.1.5
numpy : 1.19.2
pytz : 2020.4
dateutil : 2.8.1
pip : 20.3.3
setuptools : 51.0.0.post20201207
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.19.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : 0.8.7
xarray : None
xlrd : None
xlwt : None
numba : None

Activity

added
Needs TriageIssue that has not been reviewed by a pandas team member
on Dec 16, 2020
simonjayhawkins

simonjayhawkins commented on Dec 19, 2020

@simonjayhawkins
Member

Thanks @Heerozh for the report

Version 1.1.5 returned different result compared to earlier versions

first bad commit: [87e554d] BUG: RollingGroupby when groupby key is in the index (#37661) cc @mroeschke

added
RegressionFunctionality that used to work in a prior pandas version
Windowrolling, ewma, expanding
and removed
Needs TriageIssue that has not been reviewed by a pandas team member
on Dec 19, 2020
jreback

jreback commented on Dec 19, 2020

@jreback
Contributor

pls check master - i believe this is already patched
it was not simple to backport and so only fixed in 1.2

simonjayhawkins

simonjayhawkins commented on Dec 19, 2020

@simonjayhawkins
Member

not fixed in master

>>> pd.__version__
'1.3.0.dev0+100.g54682234e3'
>>>
>>> arrays = [
...     ["Falcon", "Falcon", "Parrot", "Parrot"],
...     ["Captive", "Wild", "Captive", "Wild"],
... ]
>>> index = pd.MultiIndex.from_arrays(arrays, names=("Animal", "Type"))
>>> df = pd.DataFrame({"Max Speed": [390.0, 350.0, 30.0, 20.0]}, index=index)
>>>
>>> df.groupby(level=0)["Max Speed"].rolling(2).sum()
Animal
Falcon      NaN
Falcon    740.0
Parrot      NaN
Parrot     50.0
Name: Max Speed, dtype: float64
>>>
simonjayhawkins

simonjayhawkins commented on Dec 19, 2020

@simonjayhawkins
Member

The odd thing is that the result has a MultiIndex with just one level..

>>> _.index
MultiIndex([('Falcon',),
            ('Falcon',),
            ('Parrot',),
            ('Parrot',)],
           names=['Animal'])
>>>

3 remaining items

simonjayhawkins

simonjayhawkins commented on Jan 16, 2021

@simonjayhawkins
Member

re-opening as #38737 reverted.

modified the milestones: 1.2.1, 1.2.2 on Jan 16, 2021
modified the milestones: 1.2.2, 1.2.3 on Feb 8, 2021
simonjayhawkins

simonjayhawkins commented on Feb 8, 2021

@simonjayhawkins
Member

moving to 1.2.3. dependent on discussion in #38787

modified the milestones: 1.2.3, 1.2.4 on Mar 2, 2021
modified the milestones: 1.2.4, 1.3 on Apr 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugGroupbyRegressionFunctionality that used to work in a prior pandas versionWindowrolling, ewma, expanding

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Participants

      @jreback@Heerozh@mroeschke@simonjayhawkins

      Issue actions

        BUG: MultiIndex RollingGroupby returns only one level of index · Issue #38523 · pandas-dev/pandas