Skip to content

Disable .str-accessor for byte data #23011

Closed
@h-vetinari

Description

@h-vetinari
Contributor

This supersedes #22721.

Pandas is trying to straddle many different chasms, which leads to undesirable behaviour on the fringes. For the purpose of this issue, I'm talking mainly about

  1. supporting python 2/3 (will be over soon...)
  2. being largely based on numpy's type system

From the first point, we have the inconsistent handling of str vs. bytes, so having the Series-concatenator work with bytes is a necessity in Python 2.

Mostly due to the second point, there's no proper string dtype, it's just hiding in the object dtype. I started #22721 as a side issue which came up while refactoring in #22725. Then I got told that:

We do NOT handle bytes in .str if you want to add tests and raise, pls do so, but not going to 'make it work better'. It is amazingly confusing and causes all sorts of errors. We probably don't have explicit checks on this (though I thought that we always infer on the strings that must be string/unicode and never bytes).

However, it works already -- the Series.str-accessor already checks that it can only be called on an object column, but there's not much more it can do (not least because inspecting every element of a Series would be very performance-intense). Consequently, .str.cat currently does work on bytes data, and easily at that:

>>> import pandas as pd
>>> import numpy as np
>>> s = pd.Series(np.array(list('abc'), 'S1').astype(object))
>>> t = pd.Series(np.array(list('def'), 'S1').astype(object))
>>> s.str.cat(t, sep=b'')
0    b'ad'
1    b'be'
2    b'cf'
dtype: object
>>> s.str.cat(t, sep=b',')
0    b'a,d'
1    b'b,e'
2    b'c,f'
dtype: object

Long story short - this issue supersedes #22721, and should serve as a long term goal to disable .str once Python 2 gets dropped and/or there is a string dtype.

Activity

jreback

jreback commented on Oct 6, 2018

@jreback
Contributor

we already have the capability to infer what object types are, and use this in many places in the codebase, so this is pretty easy to do. all byte operations should be disabled explicity.

In [12]: s = pd.Series(np.array(list('abc'), 'S1').astype(object))
    ...: t = pd.Series(list('def'))
    ...: 

In [13]: from pandas.api.types import infer_dtype

In [14]: infer_dtype(s)
Out[14]: 'bytes'

In [15]: infer_dtype(t)
Out[15]: 'string'
h-vetinari

h-vetinari commented on Oct 6, 2018

@h-vetinari
ContributorAuthor

@jreback

Then it should be a pretty simple fix in the __init__ of the .str-accessor to check that infer_dtype != 'bytes'... And for the list of Series in .str.cat after _get_series_list(other).

added
Error ReportingIncorrect or improved errors from pandas
StringsString extension data type and string data
on Oct 17, 2018
added this to the Contributions Welcome milestone on Oct 17, 2018
modified the milestones: Contributions Welcome, 0.25.0 on May 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    API DesignError ReportingIncorrect or improved errors from pandasStringsString extension data type and string data

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Participants

      @jreback@h-vetinari

      Issue actions

        Disable .str-accessor for byte data · Issue #23011 · pandas-dev/pandas