Description
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
(optional) I have confirmed this bug exists on the master branch of pandas.
Note This issue is very similar to Issue #8902, which is fixed in pandas==1.3.0
, but in cases where the DataFrame to concat has different columns settings as stated below, pd.concat
is still acting weird?
import pandas as pd
# pd.concat with empty DataFrame won't coerce type now
df = pd.DataFrame([{"col1": 1}, {"col1":2}])
df2 = pd.DataFrame({})
df_new = pd.concat([df,df2])
df_new.col1.dtype # dtype('int64')
# however, pd.concat with empty DataFrame with column different than the original one will still coerce type
df = pd.DataFrame([{"col1": 1}, {"col1":2}])
df2 = pd.DataFrame({}, columns=["whatever"])
df_new = pd.concat([df,df2])
df_new.col1.dtype # dtype('float64')
# pd.concat with empty DataFrame with same columns will still coerce type to `object`
df = pd.DataFrame([{"col1": 1}, {"col1":2}])
df2 = pd.DataFrame({}, columns=["col1"])
df_new = pd.concat([df,df2])
df_new.col1.dtype # dtype('object')
Problem description
pd.concat
is acting inconsistently when concatenating empty DataFrame:
If the empty DataFrame is present with same column, that column is changed to dtype('O')
;
If the empty DataFrame is present with columns different than target, the target DataFrame has its original column values of type int64
coerced to float64
;
If the empty DataFrame is present without any column, the target DataFrame keeps its dtype;
Expected Output
I suggest just not coerce values when pd.concat
an empty DataFrame...
Output of pd.show_versions()
pandas : 1.3.0
numpy : 1.19.2
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 49.6.0.post20210108
Cython : 0.29.23
pytest : 6.2.4
hypothesis : None
sphinx : 3.5.4
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : 0.9.2
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.22.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : None
odfpy : None
openpyxl : 3.0.7
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 1.4.20
tables : None
tabulate : 0.8.9
xarray : None
xlrd : 2.0.1
xlwt : None
numba : None