-
Notifications
You must be signed in to change notification settings - Fork 1k
snap - server disconnecting client for breach of protocol #10043
Copy link
Copy link
Closed
Labels
P2High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc))High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc))bugSomething isn't workingSomething isn't workingsnapsync
Description
When a node that has snap-synced serves receipt requests from an eth/70 peer, it disconnects with BREACH_OF_PROTOCOL_MALFORMED_MESSAGE_RECEIVED. The root cause is that typed receipts (EIP-1559, ACCESS_LIST, etc.) stored during snap sync are written to the database in a format that TransactionReceiptDecoder cannot read back.
Environment
- Affects nodes that completed snap sync and are then serving GetPaginatedReceiptsMessage (eth/70) requests
- Reproduced in Besu-to-Besu snap sync testing with PR Fix eth/69 receipt encoding for Frontier tx #9900
Stack trace from serving node
org.hyperledger.besu.ethereum.rlp.RLPException: Cannot enter a lists, input is fully consumed (at bytes 0-0: [])
at BytesValueRLPInput.enterList(BytesValueRLPInput.java:511)
at TransactionReceiptDecoder.decodeTypedReceiptComponents(TransactionReceiptDecoder.java:113)
at TransactionReceiptDecoder.decodeTypedReceipt(TransactionReceiptDecoder.java:88)
at TransactionReceiptDecoder.readFrom(TransactionReceiptDecoder.java:80)
at KeyValueStoragePrefixedKeyBlockchainStorage.lambda$rlpDecodeTransactionReceipts$4(line:187)
at KeyValueStoragePrefixedKeyBlockchainStorage.rlpDecodeTransactionReceipts(line:187)
at DefaultBlockchain.getTxReceipts(DefaultBlockchain.java:458)
at EthServer.constructGetPaginatedReceiptsResponse(EthServer.java:315)
more logs
{"@timestamp":"2026-03-12T05:43:42,142","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"AbstractPeerConnection","message":"Disconnecting connection -1755704946, peer 0xcd613f6a98459537... reason 0x02 BREACH_OF_PROTOCOL_MALFORMED_MESSAGE_RECEIVED Malformed message received","throwable":""}
"@timestamp":"2026-03-12T05:43:41,582","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"AbstractPeerConnection","message":"New PeerConnection ([Connection with hashCode -1755704946 inboundInitiated? false initAt 1773294221582]) established
with peer 0xcd613f6a98459537...","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,582","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"RlpxAgent","message":"Outbound connection established to peer:
0xcd613f6a984595374115b1618bf6d446afd44f477c41f4f88966ab5ae889a932326163bb088be384319fff40ebdfc88c59d441c55a72555068901a8f2540450b","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,585","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"EthProtocolManager","message":"Received status message from PeerId: 0xcd613f6a98459537... PeerReputation score: 100, timeouts: {}, useless: 0, validated?
true, disconnected? false, client: besu/v26.3-develop-11af979/linux-aarch_64/openjdk-java-21, [Connection with hashCode -1755704946 inboundInitiated? false initAt 1773294221582],
enode://cd613f6a984595374115b1618bf6d446afd44f477c41f4f88966ab5ae889a932326163bb088be384319fff40ebdfc88c59d441c55a72555068901a8f2540450b@44.220.231.189:30303, isServingSnap false, has height 0, connected for 0 ms, capabilities: [eth/69, snap/1]:
StatusMessage{code=0, size=89} with connection [Connection with hashCode -1755704946 inboundInitiated? false initAt 1773294221582]","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,585","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"EthPeers","message":"Peer PeerId: 0xcd613f6a98459537... PeerReputation score: 100, timeouts: {}, useless: 0, validated? true, disconnected? false, client:
besu/v26.3-develop-11af979/linux-aarch_64/openjdk-java-21, [Connection with hashCode -1755704946 inboundInitiated? false initAt 1773294221582],
enode://cd613f6a984595374115b1618bf6d446afd44f477c41f4f88966ab5ae889a932326163bb088be384319fff40ebdfc88c59d441c55a72555068901a8f2540450b@44.220.231.189:30303, isServingSnap false, has height 1282304, connected for 3 ms, capabilities: [eth/69, snap/1]
status exchanged","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,585","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"ChainHeadTracker","message":"Requesting chain head info from 0xcd613f6a98459537......","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,587","level":"DEBUG","thread":"EthScheduler-Services-297","class":"PeerTaskExecutor","message":"Executed peer task GetHeadersFromPeerTask against 0xcd613f6a98459537..., response code SUCCESS, retries remaining
5","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,588","level":"DEBUG","thread":"EthScheduler-Services-297","class":"ChainHeadTracker","message":"Retrieved chain head info 1282304 (0x0ff07e2807f12ed2fb74ff67bace8a7cbf450a69ef7a557b49f237c2d2dcd310) from
0xcd613f6a98459537......","throwable":""}
{"@timestamp":"2026-03-12T05:43:42,142","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"EthProtocolManager","message":"Received malformed message
0xf9102101f9101da075f24adf6df94be508d9e9cf3522d3f018eefcebd56b2423d5b96941728fc19aa02153ea8636b63f70350feb38d9afbba
...
(BREACH_OF_PROTOCOL), disconnecting: PeerId:
0xcd613f6a98459537... PeerReputation score: 102, timeouts: {}, useless: 0, validated? true, disconnected? false, client: besu/v26.3-develop-11af979/linux-aarch_64/openjdk-java-21, [Connection with hashCode -1755704946 inboundInitiated? false initAt
1773294221582], enode://cd613f6a984595374115b1618bf6d446afd44f477c41f4f88966ab5ae889a932326163bb088be384319fff40ebdfc88c59d441c55a72555068901a8f2540450b@44.220.231.189:30303, isServingSnap false, has height 1282304, connected for 560 ms, capabilities:
[eth/69, snap/1], org.hyperledger.besu.ethereum.rlp.RLPException: Cannot enter a lists, input is fully consumed (at bytes 0-0: [])","throwable":""}
{"@timestamp":"2026-03-12T05:43:42,142","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"AbstractPeerConnection","message":"Disconnecting connection -1755704946, peer 0xcd613f6a98459537... reason 0x02
BREACH_OF_PROTOCOL_MALFORMED_MESSAGE_RECEIVED Malformed message received","throwable":""}
{"@timestamp":"2026-03-12T05:43:42,142","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"EthPeers","message":"Disconnected EthPeer 0xcd613f6a98459537...","throwable":""}
{"@timestamp":"2026-03-12T05:43:42,142","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"EthProtocolManager","message":"Disconnect - active Connection? true - Outbound - 0x02 BREACH_OF_PROTOCOL_MALFORMED_MESSAGE_RECEIVED Malformed message
received - 0xcd613f6a98459537... besu/v26.3-develop-11af979/linux-aarch_64/openjdk-java-21 - 24 peers left","throwable":""}
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
P2High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc))High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc))bugSomething isn't workingSomething isn't workingsnapsync