snap - client disconnects server for NO_RESULTS_RETURNED when close to sync

[macfarla]

occasionally

{"@timestamp":"2026-03-12T02:48:50,222","level":"DEBUG","thread":"EthScheduler-Services-68","class":"GetHeadersFromPeerTask","message":"No blockheaders returned for query starting at 0xbc879412c7f9a79df82c2ddccf22b716ea2f099bd9b37a57bf600b5ef0bb0d5d","throwable":""}
{"@timestamp":"2026-03-12T02:48:50,222","level":"DEBUG","thread":"EthScheduler-Services-68","class":"PeerTaskExecutor","message":"Invalid response NO_RESULTS_RETURNED found for GetHeadersFromPeerTask from peer 0xbfae3dad13a9b588...","throwable":""}

grep -E "02:48:50|02:48:51" /var/log/besu/besu.log | grep -i "header\|GetBlock\|EthServer\|peer"
{"@timestamp":"2026-03-12T02:48:50,323","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #2,400,465  (bc879.....b0d5d)|   44 tx ( 77.3% parallel)| 16 ws| 15 blobs| 935.44 mwei bfee|  59,335,180 (
98.9%) gas used| 105.8ms exec| 561.05 Mgas/s| 18 peers","throwable":""}
grep "02:48:5" /var/log/besu/besu.log | grep -v "^$" | head -100
{"@timestamp":"2026-03-12T02:48:50,323","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #2,400,465  (bc879.....b0d5d)|   44 tx ( 77.3% parallel)| 16 ws| 15 blobs| 935.44 mwei bfee|  59,335,180 (
98.9%) gas used| 105.8ms exec| 561.05 Mgas/s| 18 peers","throwable":""}
{"@timestamp":"2026-03-12T02:48:50,494","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"FCU(VALID) | head: bc879.....b0d5d | safe: d1b3c.....3ffa5 | finalized: de0c5.....e87af","throwable":""}

Suspect it's a race condition, not a server bug.

Look at the timestamps:

  - 02:48:50,222 — Client requests headers for bc879...
  - 02:48:50,323 — Server imports block #2,400,465 (bc879...b0d5d)

The client was requesting a block that the server hadn't finished importing yet — it arrived 101ms later. The server returned empty headers because the block wasn't in its database at the time of the request. It was mid-engine_newPayload processing.

What actually happened

1. Client learned about hash bc879... somehow (new block announcement, or it derived it as a sync target)
2. Client immediately fired GetBlockHeaders for that hash - 5 rapid retries within 6ms
3. Server was still processing engine_newPayload for that block - storage not yet committed
4. Server returned empty each time → client penalized the peer and disconnected it
5. Server finished import at 02:48:50,323 - 100ms too late

The real problem

The client's retry behavior is too aggressive for this case. Exhausting all 5 retries and disconnecting the peer within 6ms, for a hash at the tip of the chain during active block production, is not the right response. The peer should get a back-off rather than an immediate reputation penalty leading to USELESS_PEER_USELESS_RESPONSES.

This is worth looking at in the PeerReputation / PeerTaskExecutor retry logic, specifically whether there should be a distinction between "peer has no data" and "peer might have data imminently" for very recent block hashes.

more logs - multiple occurrences of NO_RESULTS_RETURNED, but only one disconnect

{"@timestamp":"2026-03-12T02:55:11,594","level":"DEBUG","thread":"EthScheduler-Services-136","class":"PeerReputation","message":"Disconnection triggered by exceeding useless response threshold for peer 0xbfae3dad13a9b588...","throwable":""}

...

grep NO_RESULTS_RETURNED /var/log/besu/besu.log
{"@timestamp":"2026-03-12T02:48:50,222","level":"DEBUG","thread":"EthScheduler-Services-68","class":"PeerTaskExecutor","message":"Invalid response NO_RESULTS_RETURNED found for GetHeadersFromPeerTask from peer 0xbfae3dad13a9b588...","throwable":""}
{"@timestamp":"2026-03-12T02:48:50,224","level":"DEBUG","thread":"EthScheduler-Services-68","class":"PeerTaskExecutor","message":"Invalid response NO_RESULTS_RETURNED found for GetHeadersFromPeerTask from peer 0xbfae3dad13a9b588...","throwable":""}
{"@timestamp":"2026-03-12T02:48:50,225","level":"DEBUG","thread":"EthScheduler-Services-68","class":"PeerTaskExecutor","message":"Invalid response NO_RESULTS_RETURNED found for GetHeadersFromPeerTask from peer 0xbfae3dad13a9b588...","throwable":""}
{"@timestamp":"2026-03-12T02:48:50,226","level":"DEBUG","thread":"EthScheduler-Services-68","class":"PeerTaskExecutor","message":"Invalid response NO_RESULTS_RETURNED found for GetHeadersFromPeerTask from peer 0xbfae3dad13a9b588...","throwable":""}
{"@timestamp":"2026-03-12T02:48:50,228","level":"DEBUG","thread":"EthScheduler-Services-68","class":"PeerTaskExecutor","message":"Invalid response NO_RESULTS_RETURNED found for GetHeadersFromPeerTask from peer 0xbfae3dad13a9b588...","throwable":""}
{"@timestamp":"2026-03-12T02:55:11,590","level":"DEBUG","thread":"EthScheduler-Services-136","class":"PeerTaskExecutor","message":"Invalid response NO_RESULTS_RETURNED found for GetHeadersFromPeerTask from peer 0xbfae3dad13a9b588...","throwable":""}
{"@timestamp":"2026-03-12T02:55:11,591","level":"DEBUG","thread":"EthScheduler-Services-136","class":"PeerTaskExecutor","message":"Invalid response NO_RESULTS_RETURNED found for GetHeadersFromPeerTask from peer 0xbfae3dad13a9b588...","throwable":""}
{"@timestamp":"2026-03-12T02:55:11,592","level":"DEBUG","thread":"EthScheduler-Services-136","class":"PeerTaskExecutor","message":"Invalid response NO_RESULTS_RETURNED found for GetHeadersFromPeerTask from peer 0xbfae3dad13a9b588...","throwable":""}
{"@timestamp":"2026-03-12T02:55:11,593","level":"DEBUG","thread":"EthScheduler-Services-136","class":"PeerTaskExecutor","message":"Invalid response NO_RESULTS_RETURNED found for GetHeadersFromPeerTask from peer 0xbfae3dad13a9b588...","throwable":""}
{"@timestamp":"2026-03-12T02:55:11,594","level":"DEBUG","thread":"EthScheduler-Services-136","class":"PeerTaskExecutor","message":"Invalid response NO_RESULTS_RETURNED found for GetHeadersFromPeerTask from peer 0xbfae3dad13a9b588...","throwable":""}