Erigon 3.0.2: Polygon MainNet Archive: Stuck with 17 Files Left 0/Bs

[skb-hashchi]

Hey I am trying to get a 3.0.2 bor-mainnet archive node synched.

At the beginning of Ottersynch I was able to download at ~ 1Gbps. So I got through most of the 8050 files quickly. Once I got to approximately 50 files left, I basically began to stall.

[INFO] [06-14|09:10:20.838] [1/1 OtterSync] Downloading progress="(8050/8050 files) 55.71% - 3.1TB/5.6TB" time-left=999hrs:99m total-time=2m4s download-rate=0B/s completion-rate=0B/s alloc=4.7GB sys=5.3GB

[INFO] [06-14|09:10:26.231] [snapshots] no progress yet files=17 list=domain/v1-commitment.3392-3424.kv,domain/v1-commitment.3328-3392.kv,history/v1-accounts.3200-3264.v,idx/v1-storage.896-960.ef,idx/v1-storage.1728-1792.ef,...

As you can see I am currently on file 17 after three days, so I am slowly getting there.

What appears to be happening is I am struggling to find peers that have these files. When I find them though, I appear to be getting traffic shaped. I have tested to make sure this is not my local ISP traffic shaping me. I basically renamed the data directory, to start fresh, and again at the beginning of the manifest I achieve 1 Gbps throughput which tells me I am able to get throughput locally over torrent. So this appears to be that the ISP of the peer I am connected to that has the files I am looking for is rate shaping the connection. I have written a script to pull out of the erigon log when I see any progress on this file list, and you can see here that once I get data, it begins to geometrically hold down the throughput so I stay connected to the peer, but my throughput is nothing.

[2025-06-14 08:17:53] [monitor] ✅ Progress detected — rate=102.4KB/s, total-time=14m19s
[2025-06-14 08:18:23] [monitor] ✅ Progress detected — rate=51.2KB/s, total-time=14m39s
[2025-06-14 08:18:54] [monitor] ✅ Progress detected — rate=3.2KB/s, total-time=15m19s
[2025-06-14 08:19:24] [monitor] ✅ Progress detected — rate=409B/s, total-time=15m39s
[2025-06-14 08:19:54] [monitor] ✅ Progress detected — rate=1B/s, total-time=16m19s

Corresponding real logs:
[INFO] [06-14|08:17:26.127] [1/1 OtterSync] Downloading progress="(8050/8050 files) 55.71% - 3.1TB/5.6TB" time-left=999hrs:99m total-time=13m59s download-rate=0B/s completion-rate=0B/s alloc=8.4GB sys=15.0GB
[INFO] [06-14|08:17:46.128] [1/1 OtterSync] Downloading progress="(8050/8050 files) 55.71% - 3.1TB/5.6TB" time-left=7200hrs:27m total-time=14m19s download-rate=102.4KB/s completion-rate=102.4KB/s alloc=9.1GB sys=15.0GB
[INFO] [06-14|08:18:06.127] [1/1 OtterSync] Downloading progress="(8050/8050 files) 55.71% - 3.1TB/5.6TB" time-left=14401hrs:2m total-time=14m39s download-rate=51.2KB/s completion-rate=51.2KB/s alloc=9.6GB sys=15.0GB
[INFO] [06-14|08:18:26.127] [1/1 OtterSync] Downloading progress="(8050/8050 files) 55.71% - 3.1TB/5.6TB" time-left=57604hrs:9m total-time=14m59s download-rate=12.8KB/s completion-rate=12.8KB/s alloc=10.0GB sys=15.0GB
[INFO] [06-14|08:18:46.188] [1/1 OtterSync] Downloading progress="(8050/8050 files) 55.71% - 3.1TB/5.6TB" time-left=230469hrs:22m total-time=15m19s download-rate=3.2KB/s completion-rate=3.2KB/s alloc=10.6GB sys=15.0GB
[INFO] [06-14|08:19:06.189] [1/1 OtterSync] Downloading progress="(8050/8050 files) 55.71% - 3.1TB/5.6TB" time-left=1846008hrs:55m total-time=15m39s download-rate=409B/s completion-rate=409B/s alloc=7.8GB sys=15.1GB
[INFO] [06-14|08:19:26.188] [1/1 OtterSync] Downloading progress="(8050/8050 files) 55.71% - 3.1TB/5.6TB" time-left=30200705hrs:53m total-time=15m59s download-rate=25B/s completion-rate=25B/s alloc=8.5GB sys=15.1GB
[INFO] [06-14|08:19:46.188] [1/1 OtterSync] Downloading progress="(8050/8050 files) 55.71% - 3.1TB/5.6TB" time-left=755017647hrs:19m total-time=16m19s download-rate=1B/s completion-rate=1B/s alloc=9.4GB sys=15.1GB

What I have done so far:

1. Looked for a Erigon 3 Archive Snaphsot that I can download. I have not been able to find one yet. (Anyone, help, thanks!)
2. I have written a script around Erigon to basically run for 30 minutes, any time it makes progress stay connected for 5 more minutes, and then restart to refresh my UDP peer connections to attempt to work around the Sandvine/DPI/Traffic Shaping software. This has gotten me from 50 down to 17 files.

Anyone have ideas?

BTW.. To note, I am also very concerned by this: progress="(8050/8050 files) 55.71% - 3.1TB/5.6TB". I am not sure how I can have only 17 files left but used only 3.6TB of the 5.6TB required. If I have some BIG files left in this 17 I will clearly never get there this way.

Many thanks!

[AskAlexSharov]

rm -rf datadir/downloader
rm datadir/snapshots/prohibit_new_downloads.lock
rm datadir/snapshots/preverified.toml
rm -rf datadir/snapshots/domain

start erigon

also i would advise to upgrade to 3.0.6
increase --torrent.download.rate=1g

[skb-hashchi]

Thank you will do.. and report back..

btw this is my current config (running this natively btw, not in docker)
#!/usr/bin/env bash

------------------------------------------------------------

Start Polygon-PoS Erigon full node (embedded rpcdaemon + trace)

------------------------------------------------------------

set -euo pipefail

ERIGON_BIN="${ERIGON_BIN:-$(dirname "$0")/build/bin/erigon}"
DATA_DIR="/Volumes/polygon/docker/polygon-stack/erigon/archive-node"
HEIMDALL_RPC="${HEIMDALL_RPC:-http://localhost:1317}"

ARGS=(
--chain bor-mainnet
--datadir "$DATA_DIR"
--prune.mode=archive
--bor.heimdall "$HEIMDALL_RPC"
--downloader.verify

RPC

--http.addr 0.0.0.0
--http.api eth,erigon,net,web3,debug,trace,txpool

Performance / limits

--torrent.download.rate 100MB
--torrent.upload.rate 20MB
--maxpeers 400
--trace.maxtraces 1000000
--db.size.limit 8TB
--state.cache 24GB # memory cache for state trie
--batchSize 2048M # execution batch
--rpc.returndata.limit 1000000
)

exec "$ERIGON_BIN" "${ARGS[@]}"

[AskAlexSharov]

Remove --downloader.verify

[skb-hashchi]

Thanks I have done all of that.
I wasn't able to figure out how to get to 3.0.6.
I got to: INFO[06-14|15:21:36.345] Build info git_branch=release/3.0 git_tag=v3.0.5-21-g8690051 git_commit=869005113458fdee0845b375a50a85a5936b6938

It buzzed through the files from 7290 down to 1267 in a couple of hours.

Seem to be stuck here now:
INFO[06-14|17:19:40.487] [1/1 OtterSync] Downloading progress="(7920/7920 files) 33.93% - 1.6TB/4.7TB" time-left=999hrs:99m total-time=1h13m20s download-rate=0B/s completion-rate=0B/s alloc=5.1GB sys=12.5GB
INFO[06-14|17:19:40.737] [snapshots] no progress yet files=1267 list=idx/v1-storage.1536-1600.ef,idx/v1-tracesfrom.256-320.ef,accessor/v1-accounts.768-832.efi,accessor/v1-logaddrs.1344-1408.efi,idx/v1-tracesto.64-128.ef,...
INFO[06-14|17:19:50.487] [p2p] GoodPeers eth67=7 eth68=41

[skb-hashchi]

I backed down to 3.0.5, and cleared all the directories / files again:
rm -rf datadir/downloader
rm datadir/snapshots/prohibit_new_downloads.lock
rm datadir/snapshots/preverified.toml
rm -rf datadir/snapshots/domain

Now it looks like I am down to 20 files.

While it says this (still 0B/s):
INFO[06-14|19:45:41.581] [1/1 OtterSync] Downloading progress="(7920/7920 files) 51.52% - 2.4TB/4.7TB" time-left=999hrs:99m total-time=53m20s download-rate=0B/s completion-rate=0B/s alloc=11.4GB sys=31.3GB
INFO[06-14|19:45:41.756] [snapshots] no progress yet files=20 list=history/v1-storage.1472-1536.v,history/v1-rcache.704-768.v,idx/v1-storage.2560-2624.ef,history/v1-accounts.896-960.v,history/v1-accounts.3264-3328.v,...

It looks like my machine is actually downloading like 320Mbps of something.. I'll continue to let it run, and update.

[skb-hashchi]

Thx @anacrolix, just to note, I started from a fresh directory and still got stuck. So it is not like I have a dirty directory or something. I actually agree with you that part of the issue is likely an issue with Cloudflare, in that I am seeing a consistent approximately 1% failure in completing file downloads. I am not quite yet sure why the downloader does not recover from that. I think it is somehow getting confused with how to handle these range requests that likely are coming back failed.

Just so you know what I did while I was waiting.

• I allowed the downloader to cycle for a couple of hours.
• parsed the debug log for all the webseed URLs. (I didn't know another way to get the full manifest)
• Wrote a quick little downloader that only used Webseed not torrents so I could troubleshoot file delivery from Cloudflare.
• It resulted in 7933 URL's, the same that the Downloader thought it needed.
• I've downloaded 5294 of them so far, with 60 failures. Those failures happen mid download, the connection just gets killed. I have to redownload sometimes 2 or 3 times to actually get the file. (again straight curl no range requests, no resume)
  Let me get through this part and see if I can get the node bootstrapped, and I'll report back results.

[anacrolix]

Oh that is interesting.

FYI fixing the downloader is a high priority item and on the cards for 3.1 (or maybe 3.2). There are some issues with CloudFlare presently that may be contributing to some difficulties.

#14649

[skb-hashchi]

Cool, yeah it is probably useful just to not worry about troubleshooting the Cloudflare issues and just make sure the code is tolerant to whatever is going on there for sure.

The real issue I see is that the entire 3 node is broken (from a bootstrap perspective) with no way to get it up until the downloader is fixed. I don't think anyone starting from scratch on 3 could get a node up.. Ah.. maybe a full node possibly would synch, but I doubt it as I got blocked early... an Archive node is definitely problematic. possibly.. Having a manual snapshot download option would be super helpful in the meantime :)

[AskAlexSharov]

Seems it was some troubles on our side try restart now

[bambuleDZ]

Thanks! We had the same problem.
Whatever you did, the downloads are working again (even without restarting the container) :-)

[skb-hashchi]

Just an interesting note, I think some of the CDN servers at Cloudflare may be misconfigured.. some respond fine, but some get this message:
curl: (56) HTTP server doesn't seem to support byte ranges. Cannot resume.

[anacrolix]

Yeah it's odd. Probably some kind of behaviour they show when things aren't working right.

[anacrolix]

Ok this is all sorted, see #15574 (reply in thread).

[skb-hashchi]

Very much appreciate all the help @anacrolix @AskAlexSharov