`Peer count low, no new peers discovered` always

[muhitrhn]

Here's my nimbus start command in a docker -

sudo docker run -d --network=host \
        -v "$JWT_FILE:/jwt.hex" \
        -v "$NIMBUS_DATA_DIR:/data" \
        "$NIMBUS_IMAGE" \
        --network=sepolia \
        --data-dir=/data \
        --jwt-secret=/jwt.hex \
        --listen-address=0.0.0.0 \
        --tcp-port=9000 \
        --udp-port=9000 \
        --external-beacon-api-url=https://lodestar-sepolia.chainsafe.io \
        --trusted-block-root=0xb9e01bc6ed0cc00bef826887d05768f1acb1114b21f0c14e5fdadd39d510e1a7 \
        --bootstrap-node=enr:-Ku4QDZ_rCowZFsozeWr60WwLgOfHzv1Fz2cuMvJqN5iJzLxKtVjoIURY42X_YTokMi3IGstW5v32uSYZyGUXj9Q_IECh2F0dG5ldHOIAAAAAAAAAACEZXRoMpCo_ujukAAAaf__________gmlkgnY0gmlwhIpEe5iJc2VjcDI1NmsxoQNHTpFdaNSCEWiN_QqT396nb0PzcUpLe3OVtLph-AciBYN1ZHCCIy0,enr:-Ku4QHRyRwEPT7s0XLYzJ_EeeWvZTXBQb4UCGy1F_3m-YtCNTtDlGsCMr4UTgo4uR89pv11uM-xq4w6GKfKhqU31hTgCh2F0dG5ldHOIAAAAAAAAAACEZXRoMpCo_ujukAAAaf__________gmlkgnY0gmlwhIrFM7WJc2VjcDI1NmsxoQI4diTwChN3zAAkarf7smOHCdFb1q3DSwdiQ_Lc_FdzFIN1ZHCCIy0,enr:-Ku4QOkvvf0u5Hg4-HhY-SJmEyft77G5h3rUM8VF_e-Hag5cAma3jtmFoX4WElLAqdILCA-UWFRN1ZCDJJVuEHrFeLkDh2F0dG5ldHOIAAAAAAAAAACEZXRoMpCo_ujukAAAaf__________gmlkgnY0gmlwhJK-AWeJc2VjcDI1NmsxoQLFcT5VE_NMiIC8Ll7GypWDnQ4UEmuzD7hF_Hf4veDJwIN1ZHCCIy0,enr:-Ku4QH6tYsHKITYeHUu5kdfXgEZWI18EWk_2RtGOn1jBPlx2UlS_uF3Pm5Dx7tnjOvla_zs-wwlPgjnEOcQDWXey51QCh2F0dG5ldHOIAAAAAAAAAACEZXRoMpCo_ujukAAAaf__________gmlkgnY0gmlwhIs7Mc6Jc2VjcDI1NmsxoQIET4Mlv9YzhrYhX_H9D7aWMemUrvki6W4J2Qo0YmFMp4N1ZHCCIy0,enr:-Ku4QDmz-4c1InchGitsgNk4qzorWMiFUoaPJT4G0IiF8r2UaevrekND1o7fdoftNucirj7sFFTTn2-JdC2Ej0p1Mn8Ch2F0dG5ldHOIAAAAAAAAAACEZXRoMpCo_ujukAAAaf__________gmlkgnY0gmlwhKpA-liJc2VjcDI1NmsxoQMpHP5U1DK8O_JQU6FadmWbE42qEdcGlllR8HcSkkfWq4N1ZHCCIy0,enr:-KO4QP7MmB3juk8rUjJHcUoxZDU9Np4FlW0HyDEGIjSO7GD9PbSsabu7713cWSUWKDkxIypIXg1A-6lG7ySRGOMZHeGCAmuEZXRoMpDTH2GRkAAAc___________gmlkgnY0gmlwhBSoyGOJc2VjcDI1NmsxoQNta5b_bexSSwwrGW2Re24MjfMntzFd0f2SAxQtMj3ueYN0Y3CCIyiDdWRwgiMo,enr:-KG4QJejf8KVtMeAPWFhN_P0c4efuwu1pZHELTveiXUeim6nKYcYcMIQpGxxdgT2Xp9h-M5pr9gn2NbbwEAtxzu50Y8BgmlkgnY0gmlwhEEVkQCDaXA2kCoBBPnAEJg4AAAAAAAAAAGJc2VjcDI1NmsxoQLEh_eVvk07AQABvLkTGBQTrrIOQkzouMgSBtNHIRUxOIN1ZHCCIyiEdWRwNoIjKA,enr:-Iq4QMCTfIMXnow27baRUb35Q8iiFHSIDBJh6hQM5Axohhf4b6Kr_cOCu0htQ5WvVqKvFgY28893DHAg8gnBAXsAVqmGAX53x8JggmlkgnY0gmlwhLKAlv6Jc2VjcDI1NmsxoQK6S-Cii_KmfFdUJL2TANL3ksaKUnNXvTCv1tLwXs0QgIN1ZHCCIyk,enr:-L64QC9Hhov4DhQ7mRukTOz4_jHm4DHlGL726NWH4ojH1wFgEwSin_6H95Gs6nW2fktTWbPachHJ6rUFu0iJNgA0SB2CARqHYXR0bmV0c4j__________4RldGgykDb6UBOQAABx__________-CaWSCdjSCaXCEA-2vzolzZWNwMjU2azGhA17lsUg60R776rauYMdrAz383UUgESoaHEzMkvm4K6k6iHN5bmNuZXRzD4N0Y3CCIyiDdWRwgiMo \
        --enr-auto-update=true \
        --nat=extip:<my-ip> \
        --rest \
        --rest-address=0.0.0.0 \
        --rest-port=4000 \
        --history=prune \
        --el=http://localhost:8551

Here's the log for nimbus (after 2 hours of running nimbus, backfill still stuck at 98.89%) -

pt" epoch=239133
WRN 2025-05-18 09:33:38.707+00:00 Peer count low, no new peers discovered    topics="networking" discovered_nodes=10 new_peers=@[] current_peers=4 wanted_peers=160
INF 2025-05-18 09:33:46.085+00:00 Slot end                                   topics="beacnde" slot=7652268 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:33:48.000+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=1ms106us123ns finalized=239126:6ad52bff peers=5 slot=7652269 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
INF 2025-05-18 09:33:58.239+00:00 Slot end                                   topics="beacnde" slot=7652269 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:33:59.999+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=232us598ns finalized=239126:6ad52bff peers=3 slot=7652270 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
WRN 2025-05-18 09:34:00.374+00:00 Peer count low, no new peers discovered    topics="networking" discovered_nodes=2 new_peers=@[] current_peers=3 wanted_peers=160
INF 2025-05-18 09:34:10.114+00:00 Slot end                                   topics="beacnde" slot=7652270 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:34:12.000+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=883us324ns finalized=239126:6ad52bff peers=5 slot=7652271 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
WRN 2025-05-18 09:34:17.010+00:00 Peer count low, no new peers discovered    topics="networking" discovered_nodes=3 new_peers=@[] current_peers=3 wanted_peers=160
INF 2025-05-18 09:34:22.072+00:00 Slot end                                   topics="beacnde" slot=7652271 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:34:24.000+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=867us127ns finalized=239126:6ad52bff peers=3 slot=7652272 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
WRN 2025-05-18 09:34:31.848+00:00 Peer count low, no new peers discovered    topics="networking" discovered_nodes=3 new_peers=@[] current_peers=7 wanted_peers=160
INF 2025-05-18 09:34:34.105+00:00 Slot end                                   topics="beacnde" slot=7652272 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:34:35.999+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=390us470ns finalized=239126:6ad52bff peers=4 slot=7652273 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
WRN 2025-05-18 09:34:39.435+00:00 Peer count low, no new peers discovered    topics="networking" discovered_nodes=6 new_peers=@[] current_peers=4 wanted_peers=160
INF 2025-05-18 09:34:46.122+00:00 Slot end                                   topics="beacnde" slot=7652273 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:34:48.000+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=657us643ns finalized=239126:6ad52bff peers=3 slot=7652274 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
WRN 2025-05-18 09:34:56.918+00:00 Peer count low, no new peers discovered    topics="networking" discovered_nodes=2 new_peers=@[] current_peers=4 wanted_peers=160
INF 2025-05-18 09:34:58.071+00:00 Slot end                                   topics="beacnde" slot=7652274 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:35:00.000+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=706us495ns finalized=239126:6ad52bff peers=4 slot=7652275 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
INF 2025-05-18 09:35:10.067+00:00 Slot end                                   topics="beacnde" slot=7652275 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:35:11.999+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=235us329ns finalized=239126:6ad52bff peers=4 slot=7652276 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
INF 2025-05-18 09:35:22.098+00:00 Slot end                                   topics="beacnde" slot=7652276 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:35:23.999+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=211us873ns finalized=239126:6ad52bff peers=3 slot=7652277 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
WRN 2025-05-18 09:35:27.223+00:00 Peer count low, no new peers discovered    topics="networking" discovered_nodes=2 new_peers=@[] current_peers=2 wanted_peers=160
INF 2025-05-18 09:35:34.062+00:00 Slot end                                   topics="beacnde" slot=7652277 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:35:35.999+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=502us427ns finalized=239126:6ad52bff peers=4 slot=7652278 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
WRN 2025-05-18 09:35:39.685+00:00 Peer count low, no new peers discovered    topics="networking" discovered_nodes=2 new_peers=@[] current_peers=4 wanted_peers=160
INF 2025-05-18 09:35:46.069+00:00 Slot end                                   topics="beacnde" slot=7652278 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:35:48.000+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=845us759ns finalized=239126:6ad52bff peers=3 slot=7652279 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
INF 2025-05-18 09:35:58.094+00:00 Slot end                                   topics="beacnde" slot=7652279 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:35:59.999+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=37us132ns finalized=239126:6ad52bff peers=4 slot=7652280 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
INF 2025-05-18 09:36:10.160+00:00 Slot end                                   topics="beacnde" slot=7652280 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:36:12.000+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=871us894ns finalized=239126:6ad52bff peers=5 slot=7652281 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
WRN 2025-05-18 09:36:16.192+00:00 Peer count low, no new peers discovered    topics="networking" discovered_nodes=5 new_peers=@[] current_peers=6 wanted_peers=160
INF 2025-05-18 09:36:22.433+00:00 Database checkpointed                      topics="beacnde" dur=426ms916us320ns
INF 2025-05-18 09:36:22.433+00:00 Slot end                                   topics="beacnde" slot=7652281 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:36:24.000+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=855us356ns finalized=239126:6ad52bff peers=3 slot=7652282 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
INF 2025-05-18 09:36:34.427+00:00 Database checkpointed                      topics="beacnde" dur=420ms381us730ns
INF 2025-05-18 09:36:34.427+00:00 Slot end                                   topics="beacnde" slot=7652282 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:36:35.999+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=495us411ns finalized=239126:6ad52bff peers=4 slot=7652283 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
INF 2025-05-18 09:36:46.296+00:00 Database checkpointed                      topics="beacnde" dur=289ms335us571ns
INF 2025-05-18 09:36:46.296+00:00 Slot end                                   topics="beacnde" slot=7652283 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:36:48.000+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=992us775ns finalized=239126:6ad52bff peers=4 slot=7652284 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
INF 2025-05-18 09:36:58.427+00:00 Database checkpointed                      topics="beacnde" dur=419ms787us380ns
INF 2025-05-18 09:36:58.427+00:00 Slot end                                   topics="beacnde" slot=7652284 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119
INF 2025-05-18 09:37:00.000+00:00 Slot start                                 topics="beacnde" head=ff08b532:7652119 delay=989us579ns finalized=239126:6ad52bff peers=5 slot=7652285 sync="--h--m (98.89%) 0.0000slots/s (wwwwwwwwww:7652118)/opt" epoch=239133
INF 2025-05-18 09:37:10.137+00:00 Slot end                                   topics="beacnde" slot=7652285 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 syncCommitteeDuties=none head=ff08b532:7652119

[tersec]

Part of this is that the Sepolia bootnodes had decayed somewhat. In general, Nimbus uses https://github.com/eth-clients/sepolia/blob/main/metadata/bootstrap_nodes.yaml directly, and of those, in the version of that YAML file currently shipped:

# Teku bootnode
- enr:-KO4QP7MmB3juk8rUjJHcUoxZDU9Np4FlW0HyDEGIjSO7GD9PbSsabu7713cWSUWKDkxIypIXg1A-6lG7ySRGOMZHeGCAmuEZXRoMpDTH2GRkAAAc___________gmlkgnY0gmlwhBSoyGOJc2VjcDI1NmsxoQNta5b_bexSSwwrGW2Re24MjfMntzFd0f2SAxQtMj3ueYN0Y3CCIyiDdWRwgiMo

# Unknown
- enr:-Iq4QMCTfIMXnow27baRUb35Q8iiFHSIDBJh6hQM5Axohhf4b6Kr_cOCu0htQ5WvVqKvFgY28893DHAg8gnBAXsAVqmGAX53x8JggmlkgnY0gmlwhLKAlv6Jc2VjcDI1NmsxoQK6S-Cii_KmfFdUJL2TANL3ksaKUnNXvTCv1tLwXs0QgIN1ZHCCIyk
- enr:-L64QC9Hhov4DhQ7mRukTOz4_jHm4DHlGL726NWH4ojH1wFgEwSin_6H95Gs6nW2fktTWbPachHJ6rUFu0iJNgA0SB2CARqHYXR0bmV0c4j__________4RldGgykDb6UBOQAABx__________-CaWSCdjSCaXCEA-2vzolzZWNwMjU2azGhA17lsUg60R776rauYMdrAz383UUgESoaHEzMkvm4K6k6iHN5bmNuZXRzD4N0Y3CCIyiDdWRwgiMo

had stopped working. The status of the "Unknown" ones is still unclear, but #7168 pulls in the updated version of this upstream set of bootnodes to fix the Teku one, so there will be one more working Sepolia bootnode.

That said, that's not all of what's happening there are a couple of other points:

• Sepolia simply doesn't have as many peers as either, say, mainnet or Hoodi, or probably even Holesky; and
• during syncing, Nimbus doesn't necessarily try to hold on to the full 160 gossip target, because if's not necessarily in gossip mode, which it can enter and leave.

Can't do much about the former, but regarding the latter, you mention backfill, but the logs you show don't mention backfill. Before that, had it begun backfilling?