Cannot copy string column: table expected "Utf8" but file had "LargeUtf8" error

[oliora]

When I'm trying to copy data from local Parquet file to the database I get the following error:

type mismatch for column "bla" between table and parquet file. table expected "Utf8" but file had "LargeUtf8"

Copy command:

copy table_bla from '/path/table_bla.parquet';

Column schema:

bla              | character varying           |           |          |

Parquet file schema:

/path/table_bla.parquet | bla     | BYTE_ARRAY |             | OPTIONAL        |              | UTF8             |       |           |          | STRING

[aykut-bozkurt]

pg_parquet writes and reads from Utf8 encoded StringArray of Arrow. Any single string in the array cannot exceed 2GB. LargeUtf8 makes this limit 4GB.

I think there is some interop issue here. pg_parquet always assume the arrow string array is encoded as Utf8. (we picked Utf8 since PG already has 1GB limit for text) The file looks like written by other tools as strings are encoded as LargeUtf8.

If you have chance to write strings as Utf8, this would be a quick solution. Otherwise, pg_parquet needs to pick the correct encoding, utf8 or largeutf8, for strings before reading the file.

[oliora]

I've got this file from a 3rd party (the file is probably written with Pandas or some other analytical library) so can't simply recreate it although I can probably load and save it differently with some tools. But I know for sure that LargeUtf8 fields are pretty small and will fit into corresponding columns in my database so would be convenient if pg_parquet can convert them on the fly.