Download is not cached when using --resume-retries option

[mahdoosh1]

Description

after using --resume-retries to download a package requirement (a connection error occurs between the download, but the download finishes afterwards); i have to redownload the same requirement if the installation fails after the download.

Expected behavior

the dependency should be cached after the download finishes even if the download resumed after a connection error

pip version

25.1.1

Python version

3.12.10

OS

Android Termux (Linux localhost 6.6.30-android15-8-g957ee129519c-4k #1 SMP PREEMPT Mon Mar 31 10:06:48 UTC 2025 aarch64 Android)

How to Reproduce

1. Download a package with large dependency requirement or with slow internet connection (with --resume-retries 4)
2. Cause a connection error when the download is happening
3. Cause an error after the download (Or a keyboard interrupt, not sure if it works), preferably when installing build dependencies... for the large dependency

Output

$ pip install --resume-retries 8 vidstab
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1010)'))': /simple/vidstab/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1010)'))': /simple/vidstab/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1010)'))': /simple/vidstab/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1010)'))': /simple/vidstab/
Collecting vidstab
  Using cached vidstab-1.7.4-py2.py3-none-any.whl.metadata (16 kB)
Requirement already satisfied: numpy in /data/data/com.termux/files/usr/lib/python3.12/site-packages (from vidstab) (2.2.5)
Collecting imutils>=0.5.2 (from vidstab)
  Using cached imutils-0.5.4.tar.gz (17 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting progress (from vidstab)
  Using cached progress-1.6.tar.gz (7.8 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting matplotlib (from vidstab)
  Downloading matplotlib-3.10.3.tar.gz (34.8 MB)
     ━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.4/34.8 MB 45.7 kB/s eta 0:09:38
  WARNING: Connection timed out while downloading.
  WARNING: Attempting to resume incomplete download (8.4 MB/34.8 MB, attempt 1)
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x76968c5dc0>: Failed to establish a new connection: [Errno 7] No address associated with hostname')': /packages/26/91/d49359a21893183ed2a5b6c76bec40e0b1dcbf8ca148f864d134897cfc75/matplotlib-3.10.3.tar.gz
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x76968c68a0>: Failed to establish a new connection: [Errno 7] No address associated with hostname')': /packages/26/91/d49359a21893183ed2a5b6c76bec40e0b1dcbf8ca148f864d134897cfc75/matplotlib-3.10.3.tar.gz
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7696b70a10>: Failed to establish a new connection: [Errno 7] No address associated with hostname')': /packages/26/91/d49359a21893183ed2a5b6c76bec40e0b1dcbf8ca148f864d134897cfc75/matplotlib-3.10.3.tar.gz
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7696b709b0>: Failed to establish a new connection: [Errno 7] No address associated with hostname')': /packages/26/91/d49359a21893183ed2a5b6c76bec40e0b1dcbf8ca148f864d134897cfc75/matplotlib-3.10.3.tar.gz
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7696b70c50>: Failed to establish a new connection: [Errno 7] No address associated with hostname')': /packages/26/91/d49359a21893183ed2a5b6c76bec40e0b1dcbf8ca148f864d134897cfc75/matplotlib-3.10.3.tar.gz
  Resuming download matplotlib-3.10.3.tar.gz (8.4 MB/34.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.8/34.8 MB 114.0 kB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [11 lines of output]
      WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/meson-python/
      Collecting meson-python<0.17.0,>=0.13.1
        WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)")': /packages/91/c0/104cb6244c83fe6bc3886f144cc433db0c0c78efac5dc00e409a5a08c87d/meson_python-0.16.0-py3-none-any.whl.metadata
        WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)")': /packages/91/c0/104cb6244c83fe6bc3886f144cc433db0c0c78efac5dc00e409a5a08c87d/meson_python-0.16.0-py3-none-any.whl.metadata
        WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)")': /packages/91/c0/104cb6244c83fe6bc3886f144cc433db0c0c78efac5dc00e409a5a08c87d/meson_python-0.16.0-py3-none-any.whl.metadata
        WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)")': /packages/91/c0/104cb6244c83fe6bc3886f144cc433db0c0c78efac5dc00e409a5a08c87d/meson_python-0.16.0-py3-none-any.whl.metadata
        Downloading meson_python-0.16.0-py3-none-any.whl.metadata (4.1 kB)
      Collecting pybind11!=2.13.3,>=2.13.2
        Using cached pybind11-2.13.6-py3-none-any.whl.metadata (9.5 kB)
      ERROR: Could not find a version that satisfies the requirement setuptools_scm>=7 (from versions: none)
      ERROR: No matching distribution found for setuptools_scm>=7
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
 $ pip install setuptools_scm>=7
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)")': /packages/ab/ac/8f96ba9b4cfe3e4ea201f23f4f97165862395e9331a424ed325ae37024a8/setuptools_scm-8.3.1-py3-none-any.whl.metadata
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)")': /packages/ab/ac/8f96ba9b4cfe3e4ea201f23f4f97165862395e9331a424ed325ae37024a8/setuptools_scm-8.3.1-py3-none-any.whl.metadata
 $ pip install setuptools_scm>=7
 $ pip install setuptools_scm>=7
 $ pip install --resume-retries 8 vidstab
Collecting vidstab
  Using cached vidstab-1.7.4-py2.py3-none-any.whl.metadata (16 kB)
Requirement already satisfied: numpy in /data/data/com.termux/files/usr/lib/python3.12/site-packages (from vidstab) (2.2.5)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/imutils/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/imutils/
Collecting imutils>=0.5.2 (from vidstab)
  Using cached imutils-0.5.4.tar.gz (17 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting progress (from vidstab)
  Using cached progress-1.6.tar.gz (7.8 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting matplotlib (from vidstab)
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)")': /packages/26/91/d49359a21893183ed2a5b6c76bec40e0b1dcbf8ca148f864d134897cfc75/matplotlib-3.10.3.tar.gz
  Downloading matplotlib-3.10.3.tar.gz (34.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.8/34.8 MB 82.3 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done

i cut out the rest because it is irrelevant to the issue.

the package matplotlib got redownloaded despite being downloaded earlier on and it was not cached

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

Thanks for the report, we will take a look and try and reproduce. Please note pip is a volunteer project and it can take some time for an issue to be worked on if no one is free.

[ichard26]

Ugh, yeah, our caching stack is bypassed by the resumption logic. This is not good.

Unfortunately, manually adding a resumed download to the cache seems nontrivial since the download is, by nature of being resumed, split up across several requests Responses. They would have to be combined (and not just the data, but also the headers) before being passed to cachecontrol. It'd probably be best to implement best-effort caching of resumed downloads in the interim while we ask the cachecontrol folks if they're interested in accommodating this use-case.

[notatallshaw]

Actually, I think this is managable with existing APIs, PR incoming shortly.

[mahdoosh1]

wow, thank you for your hard work.
this was the fastest time i've reported an issue and it is being worked on without further questions.
i appreciate that.