[Python] Add API to map Arrow types (including extension types) to pandas ExtensionArray instances for to_pandas conversions

[asfimport]

With the next release of Pandas, it will be possible to define custom column types that back a pandas.Series. Thus we will not be able to cover all possible column types in the to_pandas conversion by default as we won't be aware of all extension arrays.

To enable users to create ExtensionArray instances from Arrow columns in the to_pandas conversion, we should provide a hook in the to_pandas call where they can overload the default conversion routines with the ones that produce their ExtensionArray instances.

This should avoid additional copies in the case where we would nowadays first convert the Arrow column into a default Pandas column (probably of object type) and the user would afterwards convert it to a more efficient ExtensionArray. This hook here will be especially useful when you build ExtensionArrays where the storage is backed by Arrow.

The meta-issue that tracks the implementation inside of Pandas is: pandas-dev/pandas#19696

Reporter: Uwe Korn / @xhochy
Assignee: Joris Van den Bossche / @jorisvandenbossche

Related issues:

• [Python] support pandas' nullable Integer type in from_pandas (blocks)
• [Python] Add API to map Arrow types to pandas ExtensionDtypes for to_pandas conversions (relates to)
• [Python] Interface for converting pandas ExtensionArray / other custom array objects to pyarrow Array (relates to)
• [Python] Provide Python API for creating user-defined data types that can survive Arrow IPC (depends upon)

PRs and other links:

• GitHub Pull Request #5512

_{Note: This issue was originally created as ARROW-2428. Please see the migration documentation for further details.}

[asfimport]

Alex Hagerman / @n0mn0m:
@xhochy I was reading through the meta issue and trying to understand what we have to make sure to pass. Do you think this has settled enough to begin work? It appears pandas will expect a class defining the type, which I'm guessing the objects in the arrow column will be instances of that user type? Do we expect arrow columns to meet all the requirements of ExtensionArray?

 

I was specifically looking at this to understand what options have to be passed and what the ExtensionArray requires.

https://github.com/pandas-dev/pandas/pull/19174/files#diff-e448fe09dbe8aed468d89a4c90e65cff

[asfimport]

Uwe Korn / @xhochy:
We had a stab at this at the AHL hackathon and came to the conclusion that we should wait for the Pandas 0.23 release. The interface still needs to a settle a bit more and it is not as easy as initially expected as we use (deep) internal APIs from Pandas to make the conversion from Arrow Tables to DataFrames fast. Thus I have removed the beginner label. Someone knowing the internals of Pandas very well might solve this easily but others first need to understand the mechanics of Pandas' BlockManager.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@xhochy did you have already a specific hook in mind or tried something specific at the AHL hackathon?

One way might be to allow the user to specify the target dtypes in to_pandas (on an optional per column basis). If an ExtensionDtype instance is passed there, arrow could delegate converting the arrow array to a pandas ExtensionArray to the ExtensionDtype/Array class itself.

Similarly, if we start storing the name of the ExtensionDtype in the pandas metadata, we could also automatically re-create the dtype from that name (without the need for the user to pass it explicitly, for the default).

See also the discussion in pandas-dev/pandas#20612

[asfimport]

Wes McKinney / @wesm:
I just updated the title to reflect the updated scope of handling general Arrow-to-ExtensionArray mappings, including the new C++ ExtensionType facility

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I am working on the actual ability to create ExtensionBlocks in the conversion to pandas (ARROW-6321, #5162), but to complete that work, we also need to solve this issue about how we can know / the user can indicate which columns to convert to what type.

Below I put some (long) thoughts (you can also read it / comment on google docs) about possibilities of how the API could look like to convert back to pandas ExtensionArrays. Some feedback / ideas on the API is very welcome!

Conversion arrow -> pandas

Different use cases and options:

Case 1: basic roundtrip of pandas ExtensionArrays (without involvement of arrow ExtensionTypes). For example, for pandas' nullable integer or for fletcher's arays, they map to native arrow arrays (they don't need an Arrow ExtensionType). It would be nice that DataFrames holding such pandas ExtensionArrays could roundtrip out of the box:

• When converting a DataFrame with such arrays to arrow, we save the pandas dtype in the metadata (a string representation of it). So we could use this information to know that certain columns need to be converted back to ExtensionArrays.
  The question is then: how does Arrow know in which extension array to convert it? We could look for a constructor classmethod on the pandas dtype (like PandasDtype.__constructor_from_arrow__), call that and put the returned pandas ExtensionArray in the block structure pyarrow creates. This would be kind of the inverse of __arrow_array__. Some pseudo-code to illustrate:

  {code:python}

1. the dtype name stored in the pandas metadata
   pd_dtype_name = 'Int64' # or 'fletcher[string]'
   pd_dtype = pd.api.types.pandas_dtype(pd_dtype_name)
   if hasattr(pd_dtype, 'constructor_from_arrow'):
2. indicate to ConvertTableToPandas to use ExtensionBlock for this column
   ...
   arr = ... # the pyarrow Array for this column
   ext_arr = pd_dtype.constructor_from_arrow(arr)
   block = _pd_int.make_block(ext_arr, placement=placement,
   klass=_pd_int.ExtensionBlock)
   {code}
   This will only work when the pandas dtype is registered on the pandas side (so that the string name is recognized by pandas to re-create the dtype object).

Case 2: conversion for Arrow ExtensionType. In case you defined an Arrow ExtensionType in Python, it would be nice to be able to register a default conversion to pandas. It could be this extension type that knows how to convert itself to pandas (which could either be a plain numpy array or a pandas ExtensionArray).

• We can add a method to the pyarrow.ExtensionType that converts an array of its type to a pandas-compatible array. This method can then be overridden by implementors of a ExtensionType.

• Alternatively, next to defining the pyarrow.ExtensionType, the user could register a function to be used for that type (to extend the default arrow type -> pandas type mapping).

  The method on the pyarrow.ExtensionType would be similar to the __constructor_from_arrow__ method on the pandas.ExtensionDtype. So we could also choose to let this live on the pandas dtype, and then the pyarrow.ExtensionType only needs to be mapped to a pandas dtype (but this means that the pyarrow.ExtensionType can only be converted to a pandas extension type).

Case 3: override the default conversions. There is a default mapping of pyarrow types to pandas types in the conversion. This mapping could be "extended" for ExtensionTypes (see above). But sometimes it will also be useful to override the default mapping. For example, fletcher wants to have a way to say to pyarrow: convert all your arrays to fletcher ExtensionArrays instead of the default numpy types.

• The user could register functions that extend or overrule the default arrow type -> pandas type mapping. Or it could register pandas dtypes per arrow type, and then (similar to above) that pandas dtype can know how to convert itself.
• Alternatively, the Table.to_pandas method could also gain a dtype keyword where you can specify the target dtype on a column basis.

[asfimport]

David Li / @lidavidm:
Hi Joris, overall I agree with the approach here. It's a little unfortunate that Pandas doesn't have a general column/table metadata mechanism...

I agree that we want both a default hook for ExtensionType->Pandas conversions, and a way to override conversions on an individual basis. I think adding a new argument to to_pandas is easier than maintaining yet another function registry. Similarly, adding a conversion method on ExtensionType (or maybe that should be a future ExtensionArray class?) would be preferable to maintaining a registry.

If we have something like pa.ExtensionType.\_\_pandas_array\_\_, should we also have pa.ExtensionType.\_\_pandas_dtype\_\_?

[asfimport]

Wes McKinney / @wesm:
It seems like the pandas glue can be part of the Python-side ExtensionType implementation, so a new registry is not needed (nor adding parameters to to_pandas, etc.). Is that right?

[asfimport]

David Li / @lidavidm:
It sounds like a new registry isn't needed, but adding parameters to to_pandas would be useful for customizing conversions of built-in types; Joris notes Fletcher would want to use that.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:

It seems like the pandas glue can be part of the Python-side ExtensionType implementation,

Yes, it might also need to interact with the pandas extension type, otherwise it would only solve part of the problem. I think it would be nice if it is also possible to create a pandas ExtensionArray without having a pyarrow ExtensionType (the "case 1" described above). Eg to roundtrip pandas' nullable integer type, or a future string dtype (which both don't need an arrow ExtensionType).

If we would want to support that, one option could be to go for a PandasExtensionDtype.\_\_constructor_from_arrow\_\_ that knows how to create an ExtensionArray from an arrow array, and then a pyarrow.ExtensionType.\_\_pandas_dtype\_\_ that points to the pandas extension dtype to use for this pyarrow ExtensionType

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
More thoughts on this?

I implemented a POC for case 1 described above in #5512

This allows to roundtrip pandas ExtensionArrays, assuming the the pandas.ExtensionDtype implements a \_\_from_arrow\_\_ to convert an Arrow array into a pandas ExtensionArray of that dtype (so it can be put in the resulting DataFrame as an extension array).

It doesn't yet handle to other cases described above, though.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 5512
#5512