Skip to content

DataFrame Quantile Broken with Datetime Data #23925

@WillAyd

Description

@WillAyd
Member

This works fine:

In [17]: pd.Series(pd.date_range('1/1/18', periods=5)).quantile()                                                                          
Out[17]: Timestamp('2018-01-03 00:00:00')

But the equivalent method with a DataFrame raises:

In [18]: pd.DataFrame(pd.date_range('1/1/18', periods=5)).quantile()                                                                       
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-18-68ffc067f6f0> in <module>
----> 1 pd.DataFrame(pd.date_range('1/1/18', periods=5)).quantile()

~/clones/pandas/pandas/core/frame.py in quantile(self, q, axis, numeric_only, interpolation)
   7569                                      axis=1,
   7570                                      interpolation=interpolation,
-> 7571                                      transposed=is_transposed)
   7572 
   7573         if result.ndim == 2:

~/clones/pandas/pandas/core/internals/managers.py in quantile(self, **kwargs)
    500 
    501     def quantile(self, **kwargs):
--> 502         return self.reduction('quantile', **kwargs)
    503 
    504     def setitem(self, **kwargs):

~/clones/pandas/pandas/core/internals/managers.py in reduction(self, f, axis, consolidate, transposed, **kwargs)
    473 
    474         # single block
--> 475         values = _concat._concat_compat([b.values for b in blocks])
    476 
    477         # compute the orderings of our original data

~/clones/pandas/pandas/core/dtypes/concat.py in _concat_compat(to_concat, axis)
    172                 to_concat = [x.astype('object') for x in to_concat]
    173 
--> 174     return np.concatenate(to_concat, axis=axis)
    175 
    176 

ValueError: need at least one array to concatenate

Expected Output

The DataFrame should return just as Series does

Output of pd.show_versions()

INSTALLED VERSIONS

commit: 07b0c00
python: 3.6.7.final.0
python-bits: 64
OS: Darwin
OS-release: 18.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.0.dev0+1150.g07b0c003d.dirty
pytest: 4.0.0
pip: 18.1
setuptools: 40.6.2
Cython: 0.29
numpy: 1.15.4
scipy: 1.1.0
pyarrow: 0.11.1
xarray: 0.11.0
IPython: 7.1.1
sphinx: 1.8.2
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: 3.0.1
openpyxl: 2.5.9
xlrd: 1.1.0
xlwt: 1.2.0
xlsxwriter: 1.1.2
lxml: 4.2.5
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: 1.2.14
pymysql: 0.9.2
psycopg2: None
jinja2: 2.10
s3fs: 0.1.6
fastparquet: 0.1.6
pandas_gbq: None
pandas_datareader: None
gcsfs: 0.2.0

Activity

added
DatetimeDatetime data dtype
AlgosNon-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
DataFrameDataFrame data structure
on Nov 26, 2018
qwhelan

qwhelan commented on Nov 28, 2018

@qwhelan
Contributor

It appears that DataFrame.quantile() has a default of numeric_only=True, which Series.quantile() does not have:

In [5]: pd.DataFrame(pd.date_range('1/1/18', periods=5)).quantile(numeric_only=False)
Out[5]: 0   2018-01-03
        Name: 0.5, dtype: datetime64[ns]

Looks like this is a duplicate of #18608 - I'd be in favor of resolving by deprecating this flag.

self-assigned this
on Jan 28, 2019
added this to the 1.0 milestone on Jul 20, 2019
added a commit that references this issue on Jul 24, 2019
01babb5
added a commit that references this issue on Aug 16, 2019
83b7add
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

AlgosNon-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diffBugDataFrameDataFrame data structureDatetimeDatetime data dtype

Type

No type

Projects

No projects

Relationships

None yet

    Development

    Participants

    @qwhelan@WillAyd@jreback@jbrockmendel

    Issue actions

      DataFrame Quantile Broken with Datetime Data · Issue #23925 · pandas-dev/pandas