Arbitrum warnings. It expects `required field 'v' in transaction`.

[ebadiere]

Description

As an Abritrum node operator, I want an error and warning free log so that I can focus on the application. I am using Besu as my execution engine and I am seeing the following in my arbitrum logs:

WARN [03-19|15:27:23.581] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-19|15:27:23.673] created block                            l2Block=192,032,528 l2BlockHash=ff2627..bbeaf9
INFO [03-19|15:27:24.624] rpc response                             method=eth_getTransactionByBlockHashAndIndex logId=4,501,040 err="missing required field 'v' in transaction" result={}        attempt=0 args="[\"0x3a3a66ee3f8d4230bc56eba47d07f466864312e84d967b653644d2071f68079e\", \"0x56\"]"

Transactions are being submitted so far, however these messages are a concern.

Acceptance Criteria

The issue is resolved so that the warning and error messages are not logged in Arbitrum. If this is no longer a required field and is an Arbitrum issue, let me know and I'll open an issue on their side.

Steps to Reproduce (Bug)

1. Set up a full node with Besu as the execution engine and Teku as the consensus client.
2. Set up an arbitrum node to use the node in step one.
3. Let the arbitrum node sync and watch the logs.

Expected behavior: [What you expect to happen]

Actual behavior: [What actually happens]

Frequency: [What percentage of the time does it occur?]

Logs (if a bug)

Please post relevant logs from Besu (and the consensus client, if running proof of stake) from before and after the issue.

Versions (Add all that apply)

• Software version: [besu --version] 24.3.0 Container tag hyperleger/besu:24.3.0
• Java version: [java -version]
• OS Name & Version: [cat /etc/*release]

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• Kernel Version: [uname -a]

Linux rocketpool-3 5.15.0-100-generic #110-Ubuntu SMP Wed Feb 7 13:27:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

• Virtual Machine software & version: [vmware -v] Docker
• Docker Version: [docker version]

Docker version 25.0.4, build 1a576c5

• Cloud VM, type, size: [Amazon Web Services I3-large]
  Hetzner AX52 #2323003
• Consensus Client & Version if using Proof of Stake: [e.g. Teku, Lighthouse, Prysm, Nimbus, Lodestar]

Teku
consensys/teku:24.3.0

Additional Information (Add any of the following or anything else that may be relevant)

• Besu setup info - genesis file, config options - Running Besu in rocketpool
• System info - memory, CPU
  AMD Ryzen 7 7700 Octa-Core "Raphael" (Zen4)
  64 GB DDR5 RAM

[jframe]

@ebadiere Are you using Besu as the L1 execution client in this setup?

[aytunc-tunay]

I'm experiencing the same while using consensys/teku:24.3.0 & hyperledger/besu:24.3.0

[ebadiere]

@jframe Yes, I am using it as an L1 execution client.

[dandavid3000]

Please investigate the issue since it causes troubles with Arbitrum node.

Description

After Decun upgrade, local Arbitrum node using Besu client throws a lot of warnings for missing a field. This does not happen with Geth client. The connection may flood the network somehow. This did not happen before the Decun upgrade. . Arbitrum node cannot be synced correctly.

Steps to Reproduce (Bug)

1. Run Besu node
2. Run Arbitrum node using Besu as L1

Expected behavior:
The node should run smoothly without any continuously throwing warnings

Actual behavior:
Warnings and reconnection. Arbitrum node cannot be synced correctly

Frequency:
Always

Logs (if a bug)

Besu is running smoothly. No issue from Besu client.
Arbitrum client log

WARN [03-26|15:36:47.929] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:48.939] rpc response                             method=eth_getTransactionByBlockHashAndIndex logId=929 err="missing required field 'v' in transaction" result={}       attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:48.939] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:48.964] created block                            l2Block=194,447,134 l2BlockHash=5c210b..9aa6e6
INFO [03-26|15:36:49.956] rpc response                             method=eth_getTransactionByBlockHashAndIndex logId=938 err="missing required field 'v' in transaction" result={}       attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:49.956] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:49.964] created block                            l2Block=194,447,146 l2BlockHash=e33e97..822182
INFO [03-26|15:36:50.965] created block                            l2Block=194,447,150 l2BlockHash=89de18..a32da7
INFO [03-26|15:36:50.969] rpc response                             method=eth_getTransactionByBlockHashAndIndex logId=947 err="missing required field 'v' in transaction" result={}       attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:50.969] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:51.966] created block                            l2Block=194,447,153 l2BlockHash=1337a2..27d988
INFO [03-26|15:36:51.978] rpc response                             method=eth_getTransactionByBlockHashAndIndex logId=958 err="missing required field 'v' in transaction" result={}       attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:51.978] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:52.966] created block                            l2Block=194,447,158 l2BlockHash=1e53c0..98604b
INFO [03-26|15:36:52.987] rpc response                             method=eth_getTransactionByBlockHashAndIndex logId=967 err="missing required field 'v' in transaction" result={}       attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:52.987] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:53.966] created block                            l2Block=194,447,160 l2BlockHash=0ea965..1fb963
INFO [03-26|15:36:54.000] rpc response                             method=eth_getTransactionByBlockHashAndIndex logId=976 err="missing required field 'v' in transaction" result={}       attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:54.000] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-26|15:36:54.967] created block                            l2Block=194,447,166 l2BlockHash=adf619..9c831e
INFO [03-26|15:36:55.011] rpc response                             method=eth_getTransactionByBlockHashAndIndex logId=985 err="missing required field 'v' in transaction" result={}       attempt=0 args="[\"0x4180bd0d29fec25dcb56a7dc36871504d70293e1bedb2c3b04ad0d11244bad89\", \"0x79\"]"
WARN [03-26|15:36:55.011] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"

Versions (Add all that apply)

• Software version: 24.3.0
• Java version: java 20.0.2
• OS Name & Version: Ubuntu 22.04.1 LTS
• Consensus Client & Version if using Proof of Stake: Teku

[shemnon]

The v field is deprecated, and optional, for modern TXes in the RPC. A strong argument can be made this is a bug to ARbitrum as they don't support the current spec.

Type 1 / Access List / EIP-2930 TXes -
https://github.com/ethereum/execution-apis/blob/main/src/schemas/transaction.yaml#L293-L296
Type 2 / fee market / EIP-1559 TXes -
https://github.com/ethereum/execution-apis/blob/main/src/schemas/transaction.yaml#L268-L271

Type 3 / blob / EIP-4844 TXes don't even have the field in the API spec
https://github.com/ethereum/execution-apis/blob/main/src/schemas/transaction.yaml#L237-L252

Working it in #6819 - but I need to get at least one test in before merging.

[shemnon]

@ebadiere @dandavid3000 can you try a dev build and see if the patch works?

[ebadiere]

@shemnon Sure, can you point me to the latest dev build?

[shemnon]

https://github.com/hyperledger/besu/releases/tag/develop

[ebadiere]

Docker containers would be much easier for me to test. The tar files will be a bit more time-consuming for me, so if someone else wants to confirm it that would be great. Also, I originally reported that "transactions are being submitted so far", but now my arbitrum node as stopped syncing L2Blocks, with the following in the log:

WARN [03-29|21:25:46.166] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-29|21:25:47.225] rpc response                             method=eth_getTransactionByBlockHashAndIndex logId=7026 err="missing required field 'v' in transaction" result={}       attempt=0 args="[\"0x583f20cae0f372327282fa351e1251a7d71b90a8b09a0fe71935440af6866b52\", \"0x75\"]"
WARN [03-29|21:25:47.225] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"
INFO [03-29|21:25:47.362] shutting down because of sigint
INFO [03-29|21:25:47.362] HTTP server stopped                      endpoint=[::]:8547
INFO [03-29|21:25:47.362] HTTP server stopped                      endpoint=[::]:8548
INFO [03-29|21:25:47.362] delayed sequencer: context done          err="context canceled"
INFO [03-29|21:25:47.363] Waiting background transaction indexer to exit
INFO [03-29|21:25:47.378] Writing cached state to disk             block=193,592,630 hash=f7d39e..5da887 root=36c8e2..60d884
INFO [03-29|21:25:47.413] Persisted trie from memory database      nodes=14214 size=4.75MiB time=35.277469ms gcnodes=3489 gcsize=1.38MiB gctime=8.566473ms livenodes=23482 livesize=9.10MiB
INFO [03-29|21:25:47.413] Writing cached state to disk             block=193,592,629 hash=a97686..43d0a7 root=7f6e31..e124cf
INFO [03-29|21:25:47.414] Persisted trie from memory database      nodes=189   size=75.19KiB time="912.887µs" gcnodes=0    gcsize=0.00B   gctime=0s         livenodes=23293 livesize=9.03MiB
INFO [03-29|21:25:47.414] Writing cached state to disk             block=193,592,503 hash=d94630..db5ed0 root=f46136..330bd1
INFO [03-29|21:25:47.423] Persisted trie from memory database      nodes=1893  size=713.91KiB time=8.752817ms  gcnodes=0    gcsize=0.00B   gctime=0s         livenodes=21400 livesize=8.33MiB
INFO [03-29|21:25:47.423] Writing cached state to disk             block=193,592,503 hash=d94630..db5ed0 root=f46136..330bd1
INFO [03-29|21:25:47.423] Persisted trie from memory database      nodes=0     size=0.00B     time="1.44µs"    gcnodes=0    gcsize=0.00B   gctime=0s         livenodes=21400 livesize=8.33MiB
INFO [03-29|21:25:47.423] Writing snapshot state to disk           root=f8707d..e937dd
INFO [03-29|21:25:47.423] Persisted trie from memory database      nodes=0     size=0.00B     time="1.03µs"    gcnodes=0    gcsize=0.00B   gctime=0s         livenodes=21400 livesize=8.33MiB
INFO [03-29|21:25:47.457] Blockchain stopped

It could not get passed l2Block: 193,592,630
It seems to be related to me. Can anyone confirm this? Do we expect this fix to resolved in this fix?

[shemnon]

I don't think we do dev docker distributions anymore, but @jflo may know better.

From a local repo the command ./gradlew distDocker generates local docker images. Distributing that across non-build machines is left as an exercise for the operator.

[PlasmaPower]

On the Arbitrum side, we use the go-ethereum ethclient library to access the parent chain RPC. It seems like we hadn't pulled in ethereum/go-ethereum#28564 . We'll aim to fix that in an upcoming release. Thanks for the report!

[aytunc-tunay]

I was checking the changed files in the new release https://github.com/hyperledger/besu/releases/tag/24.3.3 but I couldn't see your changes. Am I missing something or does this version not include the solution? @shemnon

[ebadiere]

I am using 24.3.3 and I am still seeing the same behavior. Am I using the wrong version?

INFO [04-21|23:33:32.612] rpc response                             method=eth_getTransactionByBlockHashAndIndex logId=1,449,012 err="missing required field 'v' in transaction" result={}       attempt=0 args="[\"0x583f20cae0f372327282fa351e1251a7d71b90a8b09a0fe71935440af6866b52\", \"0x75\"]"
WARN [04-21|23:33:32.612] error reading inbox                      err="failed to fetch batch mentioned by batch posting report: missing required field 'v' in transaction"