Open
Description
Code Sample
>>> df = pandas.read_sql_table('table_name', 'postgres:///db_name')
>>> df.head(1)
Problem description
I used pandas.read_sql_table()
to fetch a table, in which there is a JSON column. In the returned dataframe, the double quotations wrapping the keys and values are all changed to single quotations.
Example of returned dataframe:
id | name |
---|---|
1 | {'first': 'Anna', 'last': 'Smith'} |
However, JSON standards requires double quotation, on top of that, other Python package like json
doesn't recognize JSON with single quotations.
Expected Output
id | name |
---|---|
1 | {"first": "Anna", "last": "Smith"} |
Output of pd.show_versions()
INSTALLED VERSIONS
------------------
commit : None
python : 3.7.3.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None
pandas : 0.25.3
numpy : 1.17.4
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3.1
setuptools : 42.0.2.post20191203
Cython : 0.29.14
pytest : 5.3.1
hypothesis : None
sphinx : 2.2.2
blosc : None
feather : None
xlsxwriter : 1.2.6
lxml.etree : 4.4.2
html5lib : 1.0.1
pymysql : None
psycopg2 : 2.7.6.1 (dt dec pq3 ext lo64)
jinja2 : 2.10.3
IPython : 7.9.0
pandas_datareader: None
bs4 : 4.7.1
bottleneck : 1.3.1
fastparquet : None
gcsfs : None
lxml.etree : 4.4.2
matplotlib : 3.1.1
numexpr : 2.7.0
odfpy : None
openpyxl : 3.0.2
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : 0.4.0
scipy : 1.3.1
sqlalchemy : 1.3.11
tables : 3.6.1
xarray : None
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.2.6
Metadata
Metadata
Assignees
Type
Projects
Milestone
Relationships
Development
No branches or pull requests
Activity
maxcorbeau commentedon Jan 24, 2022
Same issue here.
pd.show_versions()
INSTALLED VERSIONS
commit : 66e3805
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.13.0-27-generic
Version : #29~20.04.1-Ubuntu SMP Fri Jan 14 00:32:30 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.3.5
numpy : 1.22.1
pytz : 2021.3
dateutil : 2.8.2
pip : 21.3.1
setuptools : 60.5.0
Cython : 0.29.26
pytest : 6.2.5
hypothesis : None
sphinx : 1.8.5
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.7.1
html5lib : 1.1
pymysql : None
psycopg2 : 2.9.2 (dt dec pq3 ext lo64)
jinja2 : 3.0.3
IPython : 8.0.1
pandas_datareader: None
bs4 : 4.10.0
bottleneck : 1.3.2
fsspec : 2022.01.0
fastparquet : None
gcsfs : None
matplotlib : 3.5.1
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : 6.0.1
pyxlsb : None
s3fs : 2022.01.0
scipy : 1.7.3
sqlalchemy : 1.4.29
tables : 3.6.1
tabulate : 0.8.9
xarray : 0.17.0
xlrd : 2.0.1
xlwt : 1.3.0
numba : None
datascientist-dribe commentedon Jun 22, 2023
im facing the same issue unfortunately.
mathieulebasic commentedon Nov 14, 2023
I stumbled upon this issue as well 😢
brentshulman-silkline commentedon Nov 14, 2023
I solved it, temporarily, by using https://docs.python.org/3/library/ast.html#ast.literal_eval
loganthomas commentedon Sep 24, 2024
+1