BUG: ensure_string_array may change the array inplace

[Itayazolay]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import numpy as np
import pickle
from pandas._libs.lib import ensure_string_array
t = np.array([None, None, {'a': 1.235}])
tp = pickle.loads(pickle.dumps(t))
g = np.asarray(tp, dtype='object')

assert g is not tp
assert np.shares_memory(g, tp)

print(tp) # Expected [None, None, {'a': 1.235}]
print(ensure_string_array(tp)) # got [nan, nan, "{'a': 1.235}"]
print(tp) # Expected [None, None, {'a': 1.235}], got [nan, nan, "{'a': 1.235}"]

Issue Description

I've had an issue where df.astype(str) actually changed df in-place instead of only making a copy.
I've looked into where the values mutate and I found out that it originates in the cython implementation of ensure_string_array.

• the validation on result is only using arr is not result to determine whether to copy the data
• while this condition may be false for result = np.asarray(arr, dtype='object') - it may still share memory
  verified with np.shares_memory(arr, result)
• I suggest to add/alter the condition with np.shares_memory.

I tried to make a code that replicates the issue with the astype() example, but I didn't manage to do it easily.
I can also make a PR if the solution sounds good to you.

BTW, I'm not sure why the pickle loads/dumps change the behaviour here, but the example doesn't work without it

Expected Behavior

astype() should return a copy, if not specified otherwise
ensure_string_array() should return a copy, if not specified otherwise

Installed Versions

INSTALLED VERSIONS

commit : 37ea63d
python : 3.10.11.final.0
python-bits : 64
OS : Linux
OS-release : 6.2.0-26-generic
Version : #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 16:27:29 UTC 2
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.0.1
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 57.5.0
pip : 22.3.1
Cython : 0.29.36
pytest : 7.3.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : 4.9.3
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.13.2
pandas_datareader: None
bs4 : 4.12.2
bottleneck : 1.3.7
brotli :
fastparquet : None
fsspec : 2023.6.0
gcsfs : 2023.6.0
matplotlib : 3.7.2
numba : 0.57.1
numexpr : 2.8.4
odfpy : None
openpyxl : 3.1.2
pandas_gbq : 0.17.9
pyarrow : 9.0.0
pyreadstat : None
pyxlsb : 1.0.10
s3fs : None
scipy : 1.10.1
snappy :
sqlalchemy : None
tables : 3.8.0
tabulate : None
xarray : 2023.6.0
xlrd : 2.0.1
zstandard : 0.21.0
tzdata : 2023.3
qtpy : None
pyqt5 : None

[jorisvandenbossche]

We had a similar issue some time ago, fixed in #51073, but so that still relies on the arr is not result identity check.

This seems to be caused by a peculiarity (or bug?) in numpy related to how the object dtype array comes back from pickle. Using your code example from above, the object dtype instance from the unpickled array is different from the "default" instance:

>>> t.dtype is np.dtype("object")
True
>>> tp.dtype is np.dtype("object")
False

And then I assume that np.asarray(.., dtype="object") uses such a check above, and therefore returns the original array in case of t but returns a new object in case of tp.

[Itayazolay]

According to np.asarray docs, if order and dtype are the same as the original array, it does not copy and return the original array.
It does sound like a bug in asarray, or at least it's not entirely documented.
I'll open an issue in numpy to see what they think there.
Assuming it is not a bug, in order to fix this particular issue, It's possible to use np.shares_memory to trigger copy (can be costly) or np.may_share_memory - which should be faster but return some false positives
What do you think?

[rkern]

np.may_share_memory(result, arr) or np.shares_memory(result, arr, max_work=np.MAY_SHARE_BOUNDS) should be safe and fast for this use case. In the cases that np.asarray() does make a copy of the data, it will not be in overlapping bounds.

[Itayazolay]

Great, I'll open a PR

[Itayazolay]

take