Skip to content

snap - server disconnecting client for breach of protocol #10043

@macfarla

Description

@macfarla

When a node that has snap-synced serves receipt requests from an eth/70 peer, it disconnects with BREACH_OF_PROTOCOL_MALFORMED_MESSAGE_RECEIVED. The root cause is that typed receipts (EIP-1559, ACCESS_LIST, etc.) stored during snap sync are written to the database in a format that TransactionReceiptDecoder cannot read back.

Environment

Stack trace from serving node

  org.hyperledger.besu.ethereum.rlp.RLPException: Cannot enter a lists, input is fully consumed (at bytes 0-0: [])
    at BytesValueRLPInput.enterList(BytesValueRLPInput.java:511)
    at TransactionReceiptDecoder.decodeTypedReceiptComponents(TransactionReceiptDecoder.java:113)
    at TransactionReceiptDecoder.decodeTypedReceipt(TransactionReceiptDecoder.java:88)
    at TransactionReceiptDecoder.readFrom(TransactionReceiptDecoder.java:80)
    at KeyValueStoragePrefixedKeyBlockchainStorage.lambda$rlpDecodeTransactionReceipts$4(line:187)
    at KeyValueStoragePrefixedKeyBlockchainStorage.rlpDecodeTransactionReceipts(line:187)
    at DefaultBlockchain.getTxReceipts(DefaultBlockchain.java:458)
    at EthServer.constructGetPaginatedReceiptsResponse(EthServer.java:315)

more logs

{"@timestamp":"2026-03-12T05:43:42,142","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"AbstractPeerConnection","message":"Disconnecting connection -1755704946, peer 0xcd613f6a98459537... reason 0x02 BREACH_OF_PROTOCOL_MALFORMED_MESSAGE_RECEIVED Malformed message received","throwable":""}
"@timestamp":"2026-03-12T05:43:41,582","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"AbstractPeerConnection","message":"New PeerConnection ([Connection with hashCode -1755704946 inboundInitiated? false initAt 1773294221582]) established
with peer 0xcd613f6a98459537...","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,582","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"RlpxAgent","message":"Outbound connection established to peer:
0xcd613f6a984595374115b1618bf6d446afd44f477c41f4f88966ab5ae889a932326163bb088be384319fff40ebdfc88c59d441c55a72555068901a8f2540450b","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,585","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"EthProtocolManager","message":"Received status message from PeerId: 0xcd613f6a98459537... PeerReputation score: 100, timeouts: {}, useless: 0, validated?
true, disconnected? false, client: besu/v26.3-develop-11af979/linux-aarch_64/openjdk-java-21, [Connection with hashCode -1755704946 inboundInitiated? false initAt 1773294221582],
enode://cd613f6a984595374115b1618bf6d446afd44f477c41f4f88966ab5ae889a932326163bb088be384319fff40ebdfc88c59d441c55a72555068901a8f2540450b@44.220.231.189:30303, isServingSnap false, has height 0, connected for 0 ms, capabilities: [eth/69, snap/1]:
StatusMessage{code=0, size=89} with connection [Connection with hashCode -1755704946 inboundInitiated? false initAt 1773294221582]","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,585","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"EthPeers","message":"Peer PeerId: 0xcd613f6a98459537... PeerReputation score: 100, timeouts: {}, useless: 0, validated? true, disconnected? false, client:
 besu/v26.3-develop-11af979/linux-aarch_64/openjdk-java-21, [Connection with hashCode -1755704946 inboundInitiated? false initAt 1773294221582],
enode://cd613f6a984595374115b1618bf6d446afd44f477c41f4f88966ab5ae889a932326163bb088be384319fff40ebdfc88c59d441c55a72555068901a8f2540450b@44.220.231.189:30303, isServingSnap false, has height 1282304, connected for 3 ms, capabilities: [eth/69, snap/1]
status exchanged","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,585","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"ChainHeadTracker","message":"Requesting chain head info from 0xcd613f6a98459537......","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,587","level":"DEBUG","thread":"EthScheduler-Services-297","class":"PeerTaskExecutor","message":"Executed peer task GetHeadersFromPeerTask against 0xcd613f6a98459537..., response code SUCCESS, retries remaining
5","throwable":""}
{"@timestamp":"2026-03-12T05:43:41,588","level":"DEBUG","thread":"EthScheduler-Services-297","class":"ChainHeadTracker","message":"Retrieved chain head info 1282304 (0x0ff07e2807f12ed2fb74ff67bace8a7cbf450a69ef7a557b49f237c2d2dcd310) from
0xcd613f6a98459537......","throwable":""}
{"@timestamp":"2026-03-12T05:43:42,142","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"EthProtocolManager","message":"Received malformed message
0xf9102101f9101da075f24adf6df94be508d9e9cf3522d3f018eefcebd56b2423d5b96941728fc19aa02153ea8636b63f70350feb38d9afbba
...
(BREACH_OF_PROTOCOL), disconnecting: PeerId:
0xcd613f6a98459537... PeerReputation score: 102, timeouts: {}, useless: 0, validated? true, disconnected? false, client: besu/v26.3-develop-11af979/linux-aarch_64/openjdk-java-21, [Connection with hashCode -1755704946 inboundInitiated? false initAt
1773294221582], enode://cd613f6a984595374115b1618bf6d446afd44f477c41f4f88966ab5ae889a932326163bb088be384319fff40ebdfc88c59d441c55a72555068901a8f2540450b@44.220.231.189:30303, isServingSnap false, has height 1282304, connected for 560 ms, capabilities:
[eth/69, snap/1], org.hyperledger.besu.ethereum.rlp.RLPException: Cannot enter a lists, input is fully consumed (at bytes 0-0: [])","throwable":""}
{"@timestamp":"2026-03-12T05:43:42,142","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"AbstractPeerConnection","message":"Disconnecting connection -1755704946, peer 0xcd613f6a98459537... reason 0x02
BREACH_OF_PROTOCOL_MALFORMED_MESSAGE_RECEIVED Malformed message received","throwable":""}
{"@timestamp":"2026-03-12T05:43:42,142","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"EthPeers","message":"Disconnected EthPeer 0xcd613f6a98459537...","throwable":""}
{"@timestamp":"2026-03-12T05:43:42,142","level":"DEBUG","thread":"nioEventLoopGroup-3-4","class":"EthProtocolManager","message":"Disconnect - active Connection? true - Outbound - 0x02 BREACH_OF_PROTOCOL_MALFORMED_MESSAGE_RECEIVED Malformed message
received - 0xcd613f6a98459537... besu/v26.3-develop-11af979/linux-aarch_64/openjdk-java-21 - 24 peers left","throwable":""}

Metadata

Metadata

Assignees

Labels

P2High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc))bugSomething isn't workingsnapsync

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions