Description
This supersedes #22721.
Pandas is trying to straddle many different chasms, which leads to undesirable behaviour on the fringes. For the purpose of this issue, I'm talking mainly about
- supporting python 2/3 (will be over soon...)
- being largely based on numpy's type system
From the first point, we have the inconsistent handling of str vs. bytes, so having the Series-concatenator work with bytes is a necessity in Python 2.
Mostly due to the second point, there's no proper string dtype, it's just hiding in the object
dtype. I started #22721 as a side issue which came up while refactoring in #22725. Then I got told that:
We do NOT handle bytes in
.str
if you want to add tests and raise, pls do so, but not going to 'make it work better'. It is amazingly confusing and causes all sorts of errors. We probably don't have explicit checks on this (though I thought that we always infer on the strings that must be string/unicode and never bytes).
However, it works already -- the Series.str
-accessor already checks that it can only be called on an object column, but there's not much more it can do (not least because inspecting every element of a Series would be very performance-intense). Consequently, .str.cat
currently does work on bytes data, and easily at that:
>>> import pandas as pd
>>> import numpy as np
>>> s = pd.Series(np.array(list('abc'), 'S1').astype(object))
>>> t = pd.Series(np.array(list('def'), 'S1').astype(object))
>>> s.str.cat(t, sep=b'')
0 b'ad'
1 b'be'
2 b'cf'
dtype: object
>>> s.str.cat(t, sep=b',')
0 b'a,d'
1 b'b,e'
2 b'c,f'
dtype: object
Long story short - this issue supersedes #22721, and should serve as a long term goal to disable .str
once Python 2 gets dropped and/or there is a string dtype.
Activity
jreback commentedon Oct 6, 2018
we already have the capability to infer what object types are, and use this in many places in the codebase, so this is pretty easy to do. all byte operations should be disabled explicity.
h-vetinari commentedon Oct 6, 2018
@jreback
Then it should be a pretty simple fix in the
__init__
of the.str
-accessor to check thatinfer_dtype != 'bytes'
... And for the list of Series in.str.cat
after_get_series_list(other)
.