Skip to content

nethtest: add --engineTest, --stateTest, --jsonout, --workers flags#11035

Draft
spencer-tb wants to merge 10 commits intoNethermindEth:masterfrom
spencer-tb:feat/evm-enginetest
Draft

nethtest: add --engineTest, --stateTest, --jsonout, --workers flags#11035
spencer-tb wants to merge 10 commits intoNethermindEth:masterfrom
spencer-tb:feat/evm-enginetest

Conversation

@spencer-tb
Copy link
Copy Markdown

@spencer-tb spencer-tb commented Apr 4, 2026 •
edited
Loading

Summary

Add --engineTest flag to nethtest for direct engine fixture execution, along with --stateTest, --jsonout, --workers, and --run flags. This enables running blockchain_test_engine fixtures without requiring Hive or full client startup.

nethtest --engineTest

Engine tests go through the real IEngineRpcModule via IJsonRpcService.SendRequestAsync("engine_newPayloadVX"), exercising the full Engine API code path including version validation, payload processing, and forkchoice state management. This is the same path that Hive consume engine exercises, but without Docker/client startup overhead.

--workers flag

Adds parallel file processing via Parallel.ForEachAsync to all three runners. This doesn't do much right now unless we make it more lightweight. WIP.

Other changes

  • --stateTest flag — explicitly run as state test (previously the implicit default; now required)
  • --jsonout flag — JSON array output for blocktest/enginetest (statetest already outputs JSON)
  • --run alias for --filter — matching geth/erigon naming convention
  • Replaced NUnit Assert.That in RunNewPayloads with explicit exceptions for structured error reporting

Benchmarks

Tested against EEST v5.3.0 stable fixtures.

For reference, Hive runs of the same test suite on the same Nethermind version:

  • consume engine: 16h 39m
  • consume rlp: >24h

nethtest --engineTest — exercises the same engine code paths as consume engine (40,519 tests):

Workers Time Speedup vs serial vs Hive consume engine
1 3m10s 1x ~316x
8 2m58s 1.1x ~337x

nethtest --blockTest — exercises the same execution paths as consume rlp (43,912 tests):

Workers Time Speedup vs serial vs Hive consume rlp
1 1m47s 1x >800x
8 57s 1.9x >1,500x

nethtest --stateTest (40,553 tests):

Workers Time Speedup vs serial
1 1m42s 1x
8 1m12s 1.4x

Hive parity

All 7 Hive consume engine failures for Nethermind on v5.3.0 are exception mapper issues in EELS (correct INVALID status returned, but error message string not mapped in NethermindExceptionMapper). The direct runner correctly reports these as passing since the execution result is correct. To fully test the mapper changes this will need to be integrated into consume direct via EELS.

Usage

# Engine test with parallel workers
nethtest --engineTest --workers 8 --input /path/to/blockchain_tests_engine/

# Block test with JSON output
nethtest --blockTest --jsonout --workers 8 --input /path/to/blockchain_tests/

# State test with regex filter
nethtest --stateTest --run "eip4844" --input /path/to/state_tests/

# No flag -> prints usage
nethtest --input foo.json
# "Please specify one of: --stateTest, --blockTest, or --engineTest"

Related: ethereum/go-ethereum#34650, erigontech/erigon#20315, ethereum/execution-spec-tests#2319

@spencer-tb spencer-tb force-pushed the feat/evm-enginetest branch from 4e22df4 to a5d2ab4 Compare April 4, 2026 11:59
This was referenced Apr 4, 2026
Open
Draft
@spencer-tb
test runner: add test-case level parallelism for engine/block/state t…
42f9aa3 
…ests

Parse all fixture files upfront and flatten to individual test cases
before distributing to workers, instead of parallelizing by file.

Nethermind full suite: 2m48s with 8 workers (was 2m58s file-level).
@spencer-tb spencer-tb mentioned this pull request Apr 5, 2026
Draft
@spencer-tb
test runner: optimize parallel execution and reduce allocations
ff44480 
- Pre-warm thread pool capped at CPU count (fixes starvation at w>=4)
- Parallel file parsing via Parallel.For
- UTF-8 byte span deserialization (File.ReadAllBytes + ReadOnlySpan<byte>)
- Pre-allocated results array instead of ConcurrentBag + sort
- Lightweight BlockchainTestsRunner constructor for parallel path
- Compiled Regex for filter matching
- Pre-initialize KZG once upfront

Prague engine tests: 24.1s → 18.5s at w=8 (23% faster).
Full suite: 2m48s → 1m45s at w=4 (37% faster).
This was referenced Apr 5, 2026
Draft
Draft
Draft
@spencer-tb spencer-tb mentioned this pull request Apr 5, 2026
Draft
4 tasks
@spencer-tb
test runner: per-type result schema and error reporting for consume d…
4a56632 
…irect

- Add ForkName to BlockchainTest, use raw network string (e.g. CancunToPragueAtTime15k)
- Add lastBlockHash, lastPayloadStatus to JSON output for block/engine tests
- Remove stateRoot from block/engine results (only state tests)
- Report validation error in `error` even on pass for negative tests:
  engine tests via PayloadStatus.ValidationError, block tests via
  BlockRemoved event and ValidateSuggestedBlock, state tests via
  TransactionResult.ErrorDescription
- Suggest invalid blocks (blockHeader=null) to actually test rejection
- Enables EELS consume direct exception mapping for all test types
@spencer-tb
fix: OverridableReleaseSpec.Name delegates to wrapped spec instead of…
afcb9de 
… hardcoded string
spencer-tb added a commit to spencer-tb/execution-specs that referenced this pull request Apr 8, 2026
@spencer-tb
feat(validate): new validate command for direct client binary testing
8e48ce6 
New `validate` CLI command for running EEST fixtures directly against
client EVM binaries, replacing Hive for execution correctness testing.

Usage:
  validate health                    # health check all clients
  validate engine --client geth      # engine tests
  validate state --client besu       # state tests
  validate block --client nethermind # block tests

Features:
- 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus
- Per-type Pydantic result models: StateTestResult, BlockTestResult,
  EngineTestResult with type-specific fields
- Exception matching: maps client error strings to EEST exception
  types via ExceptionMapper, verifies correct exception for every
  invalid test (--no-exception-check to disable)
- Cross-validation: lastBlockHash against fixture, lastPayloadStatus
  (VALID/INVALID) for engine tests
- validate.toml config for client binary paths with per-type overrides
  (state-bin, block-bin, engine-bin)
- Auto bin-workers and xdist tuning per client
- Bundled Frontier sanity fixtures for health checks
- Shared validate_helpers.py for validation logic

Client binary PRs:
- geth: ethereum/go-ethereum#34650
- erigon: erigontech/erigon#20315
- besu: besu-eth/besu#10184
- nethermind: NethermindEth/nethermind#11035
- reth: paradigmxyz/reth#23361
- ethrex: lambdaclass/ethrex#6445
- nimbus: status-im/nimbus-eth1#4101
- revm: bluealloy/revm#3544

Tracking issue: ethereum#2319
spencer-tb added a commit to spencer-tb/execution-specs that referenced this pull request Apr 8, 2026
@spencer-tb
feat(validate): new validate command for direct client binary testing
4603e56 
New `validate` CLI command for running EEST fixtures directly against
client EVM binaries, replacing Hive for execution correctness testing.

Usage:
  validate health                    # health check all clients
  validate engine --client geth      # engine tests
  validate state --client besu       # state tests
  validate block --client nethermind # block tests

Features:
- 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus
- Per-type Pydantic result models: StateTestResult, BlockTestResult,
  EngineTestResult with type-specific fields
- Exception matching: maps client error strings to EEST exception
  types via ExceptionMapper, verifies correct exception for every
  invalid test (--no-exception-check to disable)
- Cross-validation: lastBlockHash against fixture, lastPayloadStatus
  (VALID/INVALID) for engine tests
- validate.toml config for client binary paths with per-type overrides
  (state-bin, block-bin, engine-bin)
- Auto bin-workers and xdist tuning per client
- Bundled Frontier sanity fixtures for health checks
- Shared validate_helpers.py for validation logic

Client binary PRs:
- geth: ethereum/go-ethereum#34650
- erigon: erigontech/erigon#20315
- besu: besu-eth/besu#10184
- nethermind: NethermindEth/nethermind#11035
- reth: paradigmxyz/reth#23361
- ethrex: lambdaclass/ethrex#6445
- nimbus: status-im/nimbus-eth1#4101
- revm: bluealloy/revm#3544

Tracking issue: ethereum#2319
spencer-tb added a commit to spencer-tb/execution-specs that referenced this pull request Apr 8, 2026
@spencer-tb
feat(validate): new validate command for direct client binary testing
6c2b59f 
New `validate` CLI command for running EEST fixtures directly against
client EVM binaries, replacing Hive for execution correctness testing.

Usage:
  validate health                    # health check all clients
  validate engine --client geth      # engine tests
  validate state --client besu       # state tests
  validate block --client nethermind # block tests

Features:
- 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus
- Per-type Pydantic result models: StateTestResult, BlockTestResult,
  EngineTestResult with type-specific fields
- Exception matching: maps client error strings to EEST exception
  types via ExceptionMapper, verifies correct exception for every
  invalid test (--no-exception-check to disable)
- Cross-validation: lastBlockHash against fixture, lastPayloadStatus
  (VALID/INVALID) for engine tests
- validate.toml config for client binary paths with per-type overrides
  (state-bin, block-bin, engine-bin)
- Auto bin-workers and xdist tuning per client
- Bundled Frontier sanity fixtures for health checks
- Shared validate_helpers.py for validation logic

Client binary PRs:
- geth: ethereum/go-ethereum#34650
- erigon: erigontech/erigon#20315
- besu: besu-eth/besu#10184
- nethermind: NethermindEth/nethermind#11035
- reth: paradigmxyz/reth#23361
- ethrex: lambdaclass/ethrex#6445
- nimbus: status-im/nimbus-eth1#4101
- revm: bluealloy/revm#3544

Tracking issue: ethereum#2319
spencer-tb added a commit to spencer-tb/execution-specs that referenced this pull request Apr 8, 2026
@spencer-tb
feat(validate): new validate command for direct client binary testing
ff5d5dd 
New `validate` CLI command for running EEST fixtures directly against
client EVM binaries, replacing Hive for execution correctness testing.

Usage:
  validate health                    # health check all clients
  validate engine --client geth      # engine tests
  validate state --client besu       # state tests
  validate block --client nethermind # block tests

Features:
- 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus
- Per-type Pydantic result models: StateTestResult, BlockTestResult,
  EngineTestResult with type-specific fields
- Exception matching: maps client error strings to EEST exception
  types via ExceptionMapper, verifies correct exception for every
  invalid test (--no-exception-check to disable)
- Cross-validation: lastBlockHash against fixture, lastPayloadStatus
  (VALID/INVALID) for engine tests
- validate.toml config for client binary paths with per-type overrides
  (state-bin, block-bin, engine-bin)
- Auto bin-workers and xdist tuning per client
- Bundled Frontier sanity fixtures for health checks
- Shared validate_helpers.py for validation logic

Client binary PRs:
- geth: ethereum/go-ethereum#34650
- erigon: erigontech/erigon#20315
- besu: besu-eth/besu#10184
- nethermind: NethermindEth/nethermind#11035
- reth: paradigmxyz/reth#23361
- ethrex: lambdaclass/ethrex#6445
- nimbus: status-im/nimbus-eth1#4101
- revm: bluealloy/revm#3544

Tracking issue: ethereum#2319
spencer-tb added a commit to spencer-tb/execution-specs that referenced this pull request Apr 8, 2026
@spencer-tb
feat(validate): new validate command for direct client binary testing
bd2e809 
New `validate` CLI command for running EEST fixtures directly against
client EVM binaries, replacing Hive for execution correctness testing.

Usage:
  validate health                    # health check all clients
  validate engine --client geth      # engine tests
  validate state --client besu       # state tests
  validate block --client nethermind # block tests

Features:
- 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus
- Per-type Pydantic result models: StateTestResult, BlockTestResult,
  EngineTestResult with type-specific fields
- Exception matching: maps client error strings to EEST exception
  types via ExceptionMapper, verifies correct exception for every
  invalid test (--no-exception-check to disable)
- Cross-validation: lastBlockHash against fixture, lastPayloadStatus
  (VALID/INVALID) for engine tests
- validate.toml config for client binary paths with per-type overrides
  (state-bin, block-bin, engine-bin)
- Auto bin-workers and xdist tuning per client
- Bundled Frontier sanity fixtures for health checks
- Shared validate_helpers.py for validation logic

Client binary PRs:
- geth: ethereum/go-ethereum#34650
- erigon: erigontech/erigon#20315
- besu: besu-eth/besu#10184
- nethermind: NethermindEth/nethermind#11035
- reth: paradigmxyz/reth#23361
- ethrex: lambdaclass/ethrex#6445
- nimbus: status-im/nimbus-eth1#4101
- revm: bluealloy/revm#3544

Tracking issue: ethereum#2319
spencer-tb added a commit to spencer-tb/execution-specs that referenced this pull request Apr 8, 2026
@spencer-tb
feat(validate): new validate command for direct client binary testing
7c3c67e 
New `validate` CLI command for running EEST fixtures directly against
client EVM binaries, replacing Hive for execution correctness testing.

Usage:
  validate health                    # health check all clients
  validate engine --client geth      # engine tests
  validate state --client besu       # state tests
  validate block --client nethermind # block tests

Features:
- 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus
- Per-type Pydantic result models: StateTestResult, BlockTestResult,
  EngineTestResult with type-specific fields
- Exception matching: maps client error strings to EEST exception
  types via ExceptionMapper, verifies correct exception for every
  invalid test (--no-exception-check to disable)
- Cross-validation: lastBlockHash against fixture, lastPayloadStatus
  (VALID/INVALID) for engine tests
- validate.toml config for client binary paths with per-type overrides
  (state-bin, block-bin, engine-bin)
- Auto bin-workers and xdist tuning per client
- Bundled Frontier sanity fixtures for health checks
- Shared validate_helpers.py for validation logic

Client binary PRs:
- geth: ethereum/go-ethereum#34650
- erigon: erigontech/erigon#20315
- besu: besu-eth/besu#10184
- nethermind: NethermindEth/nethermind#11035
- reth: paradigmxyz/reth#23361
- ethrex: lambdaclass/ethrex#6445
- nimbus: status-im/nimbus-eth1#4101
- revm: bluealloy/revm#3544

Tracking issue: ethereum#2319
spencer-tb added a commit to spencer-tb/execution-specs that referenced this pull request Apr 8, 2026
@spencer-tb
feat(validate): new validate command for direct client binary testing
2c76c12 
New `validate` CLI command for running EEST fixtures directly against
client EVM binaries, replacing Hive for execution correctness testing.

Usage:
  validate health                    # health check all clients
  validate engine --client geth      # engine tests
  validate state --client besu       # state tests
  validate block --client nethermind # block tests

Features:
- 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus
- Per-type Pydantic result models: StateTestResult, BlockTestResult,
  EngineTestResult with type-specific fields
- Exception matching: maps client error strings to EEST exception
  types via ExceptionMapper, verifies correct exception for every
  invalid test (--no-exception-check to disable)
- Cross-validation: lastBlockHash against fixture, lastPayloadStatus
  (VALID/INVALID) for engine tests
- validate.toml config for client binary paths with per-type overrides
  (state-bin, block-bin, engine-bin)
- Auto bin-workers and xdist tuning per client
- Bundled Frontier sanity fixtures for health checks
- Shared validate_helpers.py for validation logic

Client binary PRs:
- geth: ethereum/go-ethereum#34650
- erigon: erigontech/erigon#20315
- besu: besu-eth/besu#10184
- nethermind: NethermindEth/nethermind#11035
- reth: paradigmxyz/reth#23361
- ethrex: lambdaclass/ethrex#6445
- nimbus: status-im/nimbus-eth1#4101
- revm: bluealloy/revm#3544

Tracking issue: ethereum#2319
spencer-tb added a commit to spencer-tb/execution-specs that referenced this pull request Apr 8, 2026
@spencer-tb
feat(validate): new validate command for direct client binary testing
2658023 
New `validate` CLI command for running EEST fixtures directly against
client EVM binaries, replacing Hive for execution correctness testing.

Usage:
  validate health                    # health check all clients
  validate engine --client geth      # engine tests
  validate state --client besu       # state tests
  validate block --client nethermind # block tests

Features:
- 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus
- Per-type Pydantic result models: StateTestResult, BlockTestResult,
  EngineTestResult with type-specific fields
- Exception matching: maps client error strings to EEST exception
  types via ExceptionMapper, verifies correct exception for every
  invalid test (--no-exception-check to disable)
- Cross-validation: lastBlockHash against fixture, lastPayloadStatus
  (VALID/INVALID) for engine tests
- validate.toml config for client binary paths with per-type overrides
  (state-bin, block-bin, engine-bin)
- Auto bin-workers and xdist tuning per client
- Bundled Frontier sanity fixtures for health checks
- Shared validate_helpers.py for validation logic

Client binary PRs:
- geth: ethereum/go-ethereum#34650
- erigon: erigontech/erigon#20315
- besu: besu-eth/besu#10184
- nethermind: NethermindEth/nethermind#11035
- reth: paradigmxyz/reth#23361
- ethrex: lambdaclass/ethrex#6445
- nimbus: status-im/nimbus-eth1#4101
- revm: bluealloy/revm#3544

Tracking issue: ethereum#2319
spencer-tb added a commit to spencer-tb/execution-specs that referenced this pull request Apr 8, 2026
@spencer-tb
feat(validate): new validate command for direct client binary testing
88d72b6 
New `validate` CLI command for running EEST fixtures directly against
client EVM binaries, replacing Hive for execution correctness testing.

Usage:
  validate health                    # health check all clients
  validate engine --client geth      # engine tests
  validate state --client besu       # state tests
  validate block --client nethermind # block tests

Features:
- 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus
- Per-type Pydantic result models: StateTestResult, BlockTestResult,
  EngineTestResult with type-specific fields
- Exception matching: maps client error strings to EEST exception
  types via ExceptionMapper, verifies correct exception for every
  invalid test (--no-exception-check to disable)
- Cross-validation: lastBlockHash against fixture, lastPayloadStatus
  (VALID/INVALID) for engine tests
- validate.toml config for client binary paths with per-type overrides
  (state-bin, block-bin, engine-bin)
- Auto bin-workers and xdist tuning per client
- Bundled Frontier sanity fixtures for health checks
- Shared validate_helpers.py for validation logic

Client binary PRs:
- geth: ethereum/go-ethereum#34650
- erigon: erigontech/erigon#20315
- besu: besu-eth/besu#10184
- nethermind: NethermindEth/nethermind#11035
- reth: paradigmxyz/reth#23361
- ethrex: lambdaclass/ethrex#6445
- nimbus: status-im/nimbus-eth1#4101
- revm: bluealloy/revm#3544

Tracking issue: ethereum#2319
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@flcl42 flcl42 Awaiting requested review from flcl42 flcl42 will be requested when the pull request is marked ready for review flcl42 is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@spencer-tb