logging cleanup for invalid blocks #10160
Description
This could be non-trivial as it might be hard to distinguish between our own bad blocks versus incoming bad blocks.
I don't think we should be throwing ERROR in AbstractBlockProcessor for these events.
We also duplicate the message across WARN and for parallel fallback too.
First one was https://beaconcha.in/slot/14022301
so genuine invalid network block.
2026-04-02 01:00:24.584
info
{"@timestamp":"2026-04-02T01:00:24,584","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #24,788,494 (4daec.....797d8)| 178 tx ( 34.8% parallel)| 16 ws| 5 blobs| 99.12 mwei bfee| 19,427,791 ( 32.4%) gas used| 184.4ms exec| 105.38 Mgas/s| 25 peers","throwable":""}
2026-04-02 01:00:24.689
info
{"@timestamp":"2026-04-02T01:00:24,689","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"FCU(VALID) | head: 4daec.....797d8 | safe: 6e5f7.....a6cb5 | finalized: 41219.....0c706","throwable":""}
2026-04-02 01:00:36.912
error
{"@timestamp":"2026-04-02T01:00:36,912","level":"ERROR","thread":"vert.x-worker-thread-0","class":"AbstractBlockProcessor","message":"failed persisting block due to stateroot mismatch; expected 0x8b9e6fd770199455cb46f421acdf29b3264b08daec5c3ead98d0c65aaf3b29a7, actual 0xcc84a029b12e0ee0f248e6e4091dd14c6303164539586f5cc257a1902ff6c84d","throwable":""}
2026-04-02 01:00:36.918
info
{"@timestamp":"2026-04-02T01:00:36,918","level":"INFO","thread":"vert.x-worker-thread-0","class":"MainnetParallelBlockProcessor","message":"Parallel transaction processing failure. Falling back to non-parallel processing for block #24788495 (0x1048d24856d90cc1754c27cd3af853f374085fa3a69160e99f3ab711b4ccb817)","throwable":""}
2026-04-02 01:00:37.299
error
{"@timestamp":"2026-04-02T01:00:37,299","level":"ERROR","thread":"vert.x-worker-thread-0","class":"AbstractBlockProcessor","message":"failed persisting block due to stateroot mismatch; expected 0x8b9e6fd770199455cb46f421acdf29b3264b08daec5c3ead98d0c65aaf3b29a7, actual 0xcc84a029b12e0ee0f248e6e4091dd14c6303164539586f5cc257a1902ff6c84d","throwable":""}
2026-04-02 01:00:37.302
info
{"@timestamp":"2026-04-02T01:00:37,302","level":"INFO","thread":"vert.x-worker-thread-0","class":"MainnetBlockValidator","message":"Invalid block 24788495 (0x1048d24856d90cc1754c27cd3af853f374085fa3a69160e99f3ab711b4ccb817): Optional[World State Root does not match expected value, header 0x8b9e6fd770199455cb46f421acdf29b3264b08daec5c3ead98d0c65aaf3b29a7 calculated 0xcc84a029b12e0ee0f248e6e4091dd14c6303164539586f5cc257a1902ff6c84d]","throwable":""}
2026-04-02 01:00:37.305
warn
{"@timestamp":"2026-04-02T01:00:37,305","level":"WARN","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Invalid new payload: number: 24788495, hash: 0x1048d24856d90cc1754c27cd3af853f374085fa3a69160e99f3ab711b4ccb817, parentHash: 0x4daec40f3960cb542dba47cba0fcd9cde7576bcece7419b45fc14bf96e6797d8, latestValidHash: 0x4daec40f3960cb542dba47cba0fcd9cde7576bcece7419b45fc14bf96e6797d8, status: INVALID, validationError: World State Root does not match expected value, header 0x8b9e6fd770199455cb46f421acdf29b3264b08daec5c3ead98d0c65aaf3b29a7 calculated 0xcc84a029b12e0ee0f248e6e4091dd14c6303164539586f5cc257a1902ff6c84d","throwable":""}
2026-04-02 01:00:48.684
info
{"@timestamp":"2026-04-02T01:00:48,684","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #24,788,495 (33f63.....f65d1)| 230 tx ( 32.6% parallel)| 16 ws| 11 blobs| 94.75 mwei bfee| 19,381,931 ( 32.3%) gas used| 213.1ms exec| 90.96 Mgas/s| 25 peers","throwable":""}
2026-04-02 01:00:49.014
info
{"@timestamp":"2026-04-02T01:00:49,014","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"FCU(VALID) | head: 33f63.....f65d1 | safe: 6e5f7.....a6cb5 | finalized: 41219.....0c706","throwable":""}
2026-04-02 01:01:00.711
info
{"@timestamp":"2026-04-02T01:01:00,711","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #24,788,496 (44129.....54db9)| 126 tx ( 38.1% parallel)| 16 ws| 5 blobs| 90.56 mwei bfee| 18,386,117 ( 30.6%) gas used| 213.5ms exec| 86.14 Mgas/s| 25 peers","throwable":""}
2026-04-02 01:01:00.791
info
{"@timestamp":"2026-04-02T01:01:00,791","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"FCU(VALID) | head: 44129.....54db9 | safe: 6e5f7.....a6cb5 | finalized: 41219.....0c706","throwable":""}
2026-04-02 01:01:12.579
info
{"@timestamp":"2026-04-02T01:01:12,579","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #24,788,497 (f4391.....d375f)| 51 tx ( 25.5% parallel)| 16 ws| 0 blobs| 86.18 mwei bfee| 13,911,909 ( 23.2%) gas used| 200.1ms exec| 69.51 Mgas/s| 25 peers","throwable":""}
2026-04-02 01:01:12.701
info
{"@timestamp":"2026-04-02T01:01:12,701","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"FCU(VALID) | head: f4391.....d375f | safe: b25b8.....f73b3 | finalized: 6e5f7.....a6cb5","throwable":""}
2026-04-02 01:01:24.388
error
{"@timestamp":"2026-04-02T01:01:24,388","level":"ERROR","thread":"vert.x-worker-thread-0","class":"AbstractBlockProcessor","message":"failed persisting block due to stateroot mismatch; expected 0xe3e063eba0c9bc5f9151d82c9ece89b02a589b889b15ef441f45917ed16656b4, actual 0xc8c4df9c7e10b8bbc67390cca38a16addc72a6ab3e648b478fd8038c57465131","throwable":""}
2026-04-02 01:01:24.389
info
{"@timestamp":"2026-04-02T01:01:24,389","level":"INFO","thread":"vert.x-worker-thread-0","class":"MainnetParallelBlockProcessor","message":"Parallel transaction processing failure. Falling back to non-parallel processing for block #24788498 (0x7ce2b93d69559a5c3d0c80993ce50e60d6bd1ae4426485576839e1fe5d547234)","throwable":""}
2026-04-02 01:01:24.820
error
{"@timestamp":"2026-04-02T01:01:24,820","level":"ERROR","thread":"vert.x-worker-thread-0","class":"AbstractBlockProcessor","message":"failed persisting block due to stateroot mismatch; expected 0xe3e063eba0c9bc5f9151d82c9ece89b02a589b889b15ef441f45917ed16656b4, actual 0xc8c4df9c7e10b8bbc67390cca38a16addc72a6ab3e648b478fd8038c57465131","throwable":""}
2026-04-02 01:01:24.821
info
{"@timestamp":"2026-04-02T01:01:24,821","level":"INFO","thread":"vert.x-worker-thread-0","class":"MainnetBlockValidator","message":"Invalid block 24788498 (0x7ce2b93d69559a5c3d0c80993ce50e60d6bd1ae4426485576839e1fe5d547234): Optional[World State Root does not match expected value, header 0xe3e063eba0c9bc5f9151d82c9ece89b02a589b889b15ef441f45917ed16656b4 calculated 0xc8c4df9c7e10b8bbc67390cca38a16addc72a6ab3e648b478fd8038c57465131]","throwable":""}
2026-04-02 01:01:37.160
info
{"@timestamp":"2026-04-02T01:01:37,160","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #24,788,498 (8a070.....92adf)| 514 tx ( 45.9% parallel)| 16 ws| 20 blobs| 80.40 mwei bfee| 34,852,959 ( 58.1%) gas used| 320.4ms exec| 108.79 Mgas/s| 25 peers","throwable":""}
2026-04-02 01:01:37.414
info
{"@timestamp":"2026-04-02T01:01:37,414","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"FCU(VALID) | head: 8a070.....92adf | safe: b25b8.....f73b3 | finalized: 6e5f7.....a6cb5","throwable":""}