Skip to content

Implement cumulus StorageWeightReclaim as wrapping transaction extension + frame system ReclaimWeight#6140

Merged
gui1117 merged 84 commits intomasterfrom
gui-storage-proof-size-reclaim-more-accurate
Jan 5, 2025
Merged

Implement cumulus StorageWeightReclaim as wrapping transaction extension + frame system ReclaimWeight#6140
gui1117 merged 84 commits intomasterfrom
gui-storage-proof-size-reclaim-more-accurate

Conversation

@gui1117
Copy link
Copy Markdown
Contributor

@gui1117 gui1117 commented Oct 19, 2024

(rebasing of #5234)

Issues:

  • Transaction extensions have weights and refund weight. So the reclaiming of unused weight must happen last in the transaction extension pipeline. Currently it is inside CheckWeight.
  • cumulus storage weight reclaim transaction extension misses the proof size of logic happening prior to itself.

Done:

  • a new storage ExtrinsicWeightReclaimed in frame-system. Any logic which attempts to do some reclaim must use this storage to avoid double reclaim.

  • a new function reclaim_weight in frame-system pallet: info and post info in arguments, read the already reclaimed weight, calculate the new unused weight from info and post info. do the more accurate reclaim if higher.

  • CheckWeight is unchanged and still reclaim the weight in post dispatch

  • ReclaimWeight is a new transaction extension in frame system. For solo chains it must be used last in the transactino extension pipeline. It does the final most accurate reclaim

  • StorageWeightReclaim is moved from cumulus primitives into its own pallet (in order to define benchmark) and is changed into a wrapping transaction extension.
    It does the recording of proof size and does the reclaim using this recording and the info and post info. So parachains don't need to use ReclaimWeight. But also if they use it, there is no bug.

    /// The TransactionExtension to the basic transaction logic.
    pub type TxExtension = cumulus_pallet_weight_reclaim::StorageWeightReclaim<
         Runtime,
         (
                 frame_system::CheckNonZeroSender<Runtime>,
                 frame_system::CheckSpecVersion<Runtime>,
                 frame_system::CheckTxVersion<Runtime>,
                 frame_system::CheckGenesis<Runtime>,
                 frame_system::CheckEra<Runtime>,
                 frame_system::CheckNonce<Runtime>,
                 frame_system::CheckWeight<Runtime>,
                 pallet_transaction_payment::ChargeTransactionPayment<Runtime>,
                 BridgeRejectObsoleteHeadersAndMessages,
                 (bridge_to_rococo_config::OnBridgeHubWestendRefundBridgeHubRococoMessages,),
                 frame_metadata_hash_extension::CheckMetadataHash<Runtime>,
         ),
    >;

@gui1117 gui1117 added T9-cumulus This PR/Issue is related to cumulus. T2-pallets This PR/Issue is related to a particular pallet. labels Oct 19, 2024
@bkchr
Copy link
Copy Markdown
Member

bkchr commented Oct 20, 2024

Anyway this doesn't have any vulnerability, it just wastes resources, but people should put the CheckWeight last in the pipeline.

I would say it depends. When you are required to put CheckWeight as the latest extension, it also means that you are missing a cheap, early return.

@georgepisaltu
Copy link
Copy Markdown
Contributor

you are missing a cheap, early return

Going through the pipeline should be cheap anyway. It's just extensions which are pretty light and the "wasted" work for overweight transactions should be done off-chain when validators are building their blocks.

Because the weight check isn't hardcoded and users can build whatever extension they like to handle it, we need to have some sort of convention when we introduce other weight related logic. I skimmed through the PR and I like the approach, but I won't formally approve because I didn't review thoroughly.

@gui1117
Copy link
Copy Markdown
Contributor Author

gui1117 commented Oct 21, 2024

Anyway this doesn't have any vulnerability, it just wastes resources, but people should put the CheckWeight last in the pipeline.

I would say it depends. When you are required to put CheckWeight as the latest extension, it also means that you are missing a cheap, early return.

Maybe it is time to split this transaction extension into CheckWeight and RefundWeight.

EDIT: or we can do the RefundWeight in note_applied_extrinsic.

EDIT: or we can use a storage to store the weight refunded by CheckWeight, then StorageWeightReclaim will just take this value instead of trying to guess it incorrectly.

EDIT: I decided to with a new storage ExtrinsicWeight or ExtrinsicWeightRefunded, CheckWeight will register its refund there, then StorageWeightReclaim will undo the CheckWeight operation and do the correct refund.

Later we can introduce another RefundWeight for solo-chains or parachains that doesn't want to use StorageWeightReclaim. RefundWeight can be placed at the end of the pipeline, and CheckWeight is unchanged and not breaking.

@gui1117 gui1117 requested a review from a team as a code owner October 24, 2024 08:50
@paritytech paritytech deleted a comment from github-actions bot Oct 25, 2024
@paritytech paritytech deleted a comment from github-actions bot Oct 25, 2024
@paritytech paritytech deleted a comment from github-actions bot Oct 25, 2024
@paritytech paritytech deleted a comment from github-actions bot Oct 25, 2024
Merged via the queue into master with commit 63c73bf Jan 5, 2025
@gui1117 gui1117 deleted the gui-storage-proof-size-reclaim-more-accurate branch January 5, 2025 04:00
ordian added a commit that referenced this pull request Jan 7, 2025
* master: (256 commits)
  fix chunk fetching network compatibility zombienet test (#6988)
  chore: delete repeat words (#7034)
  Print taplo version in CI (#7041)
  Implement cumulus StorageWeightReclaim as wrapping transaction extension + frame system ReclaimWeight (#6140)
  Make `TransactionExtension` tuple of tuple transparent for implication (#7028)
  Replace duplicated whitelist with whitelisted_storage_keys (#7024)
  [WIP] Fix networking-benchmarks (#7036)
  [docs] Fix release naming (#7032)
  migrate pallet-mixnet to umbrella crate (#6986)
  Improve remote externalities logging (#7021)
  Fix polkadot sdk doc. (#7022)
  Remove warning log from frame-omni-bencher CLI (#7020)
  [pallet-revive] fix file case (#6981)
  Add workflow for networking benchmarks (#7029)
  [CI] Skip SemVer on R0-silent and update docs (#6285)
  correct path in cumulus README (#7001)
  sync: Send already connected peers to new subscribers (#7011)
  Excluding chainlink domain for link checker CI (#6524)
  pallet-bounties: Fix benchmarks for 0 ED (#7013)
  Log peerset set ID -> protocol name mapping (#7005)
  ...
ffarall added a commit to Moonsong-Labs/storage-hub that referenced this pull request Mar 13, 2026
### Notable changes:

* Downgraded rust toolchain version from `1.91` to `1.90`

  ```toml
   # Rust 1.90 is required due to polkadot-sdk stable2503 compatibility.
   # Rust 1.91+ breaks pallet-revive-fixtures compilation with error:
   #   "target-pointer-width: invalid type: string '64', expected u16"
# This is because polkavm-linker generates target specs with
"target-pointer-width": "64" (string),
# but Rust 1.91 (PR #144443) changed this to require an integer:
"target-pointer-width": 64.
# polkadot-sdk stable2503 was tested with Rust 1.84.1; 1.90 is the
latest compatible version.
   channel = "1.90"
  ```

Changes were also caused by this change due to clippy warnings that were
not present in 1.91.

* paritytech/polkadot-sdk#7634:

`DecodeWithMemTracking` trait derivation was added to `RuntimeCall`.
~~This change would have been simply inherited if sp-trie updated their
`CompactProof` type which we use extensively in the codebase.~~

~~To satisfy the bound, proof types at extrinsic boundaries now use
`Vec<Vec<u8>>` instead of `CompactProof` (aliased as
[`CompactProofEncodedNodes`](https://github.com/Moonsong-Labs/storage-hub/pull/671/files#diff-c35139710c5462fa6b2f0c0376c749266f0edfc94da2acf58ddecab8c39de37dR27-R32).
This represents the `encoded_nodes` field of a compact proof which
implements `DecodeWithMemTracking` by the parity-scale-code crate. The
conversion to `CompactProof` is done internally when needed for trie
operations.~~

~~This was one of the two solutions that posed the least amount of code
changes and is easier to change once `CompactProof` implements the new
trait. The other more intrusive way was wrapping the `CompactProof` in a
struct which implements the trait, but this lead to many changes that
were hard to track.~~

> **Note:** This is a **temporary workaround**. Once `CompactProof`
implements `DecodeWithMemTracking` upstream in `sp-trie`, this change
will be reverted to use `CompactProof` directly again.
paritytech/polkadot-sdk#11028

* paritytech/polkadot-sdk#7043

`sp-std` crate is deprecated and all uses of it in the codebase have
been replaced with the `alloc` or `core` equivalent imports.

* paritytech/polkadot-sdk#6140

Wrapped transaction extensions for both parachain and solochain-evm
runtimes with `StorageWeightReclaim`. `StorageWeightReclaim` is meant to
be used as a wrapping of the whole transaction extension pipeline, and
will take into account all proof size accurately.

  ```rust
  /// The TransactionExtension to the basic transaction logic.
pub type TxExtension =
cumulus_pallet_weight_reclaim::StorageWeightReclaim<
    Runtime,
    (
        frame_system::CheckNonZeroSender<Runtime>,
        frame_system::CheckSpecVersion<Runtime>,
        frame_system::CheckTxVersion<Runtime>,
        frame_system::CheckGenesis<Runtime>,
        frame_system::CheckEra<Runtime>,
        frame_system::CheckNonce<Runtime>,
        frame_system::CheckWeight<Runtime>,
        pallet_transaction_payment::ChargeTransactionPayment<Runtime>,
        frame_metadata_hash_extension::CheckMetadataHash<Runtime>,
    ),
  >;
  ```

* **Frontier EVM / `ProofSizeExt` on solochain-evm:**

Frontier EVM (stable2503) introduced `get_proof_size()` host function
calls at multiple points in the EVM runner to measure actual proof size
via the `ProofSizeExt` host extension. If `ProofSizeExt` is present
during block building but absent during block import on syncing nodes,
`get_proof_size()` returns different values on each path, leading to
different gas/weight calculations, different state roots, and digest
mismatch panics.

  To fix this, two changes were made to the solochain-evm node service:

* **Block building**: `ProposerFactory::with_proof_recording` (instead
of `ProposerFactory::new`) to register `ProofSizeExt` during block
authoring.
* **Block import**: `sc_service::new_full_parts_record_import(...,
true)` (instead of `new_full_parts`) so that the Client registers
`ProofSizeExt` during block import on syncing BSP/MSP nodes, matching
the block building path.

- paritytech/polkadot-sdk#6349:

Genesis config preset files have been migrated to use the new
`build_struct_json_patch!` macro instead of the previous pattern of
constructing a full `RuntimeGenesisConfig` struct and serializing it
with `serde_json::to_value()`.

  Before:

  ```rust
  let config = RuntimeGenesisConfig {
      balances: BalancesConfig { ... },
      ..Default::default()
  };
  serde_json::to_value(config).expect("Could not build genesis config.")
  ```

  After:

  ```rust
  use frame_support::build_struct_json_patch;
  build_struct_json_patch!(RuntimeGenesisConfig {
      balances: BalancesConfig { ... },
  })
  ```

* paritytech/polkadot-sdk#8461

Substrate based chains are now configured to use `litep2p` as the
network backend (before was libp2p)

While `litep2p` implements the same wire protocol and is interoperable
with `libp2p` nodes, several behavioral differences required fixes:

* **GRANDPA notification keepalive**: In dev mode with manual sealing,
the GRANDPA voter is not started but its notification protocol is still
registered for P2P negotiation. `litep2p` kills all P2P connections if
any registered notification protocol's service handle is dropped (unlike
`libp2p` which tolerates this). A keepalive task was added that drains
GRANDPA notification events without running consensus, keeping the
protocol handler alive for the lifetime of the node.

* **Chunk uploader retry count**: `litep2p` maps transient connectivity
errors (`ConnectionClosed`, `SubstreamClosed`, `dial-failed`, etc.) to
`RequestFailure::Refused` instead of `RequestFailure::NotConnected`. The
chunk uploader's retry limit for `Refused` errors was increased from 3
to 30 to handle peer recovery scenarios during file uploads.
  
* **Integration tests**: Integration tests were modified to take into
account hew new `Refused` error propagation which changed timing and
logging assertions in the tests.

## ⚠️ Breaking Changes ⚠️


- **Short description**

The transaction extension pipeline (`SignedExtra` / `TxExtension`) must
now be wrapped with
`cumulus_pallet_weight_reclaim::StorageWeightReclaim`. This wrapper
measures PoV size before and after execution and reclaims unused storage
weight, including weight consumed by the extensions themselves. Runtimes
must also implement `cumulus_pallet_weight_reclaim::Config`.

  This affects:
  - The `SignedExtra` / `TxExtension` type alias
- The `from_minimal_extension` implementation (must wrap the inner
extension tuple)
- The `compute_signed_extra_implicit` runtime API implementation (same
wrapping)

- **Who is affected**

* **Downstream runtimes** that integrate StorageHub pallets and define
their own transaction extension pipeline
* **Client code** that implements
`ExtensionOperations::from_minimal_extension`

- **Suggested code changes**

  Type alias:

  Before:

  ```rust
  pub type SignedExtra = (
      frame_system::CheckNonZeroSender<Runtime>,
      frame_system::CheckSpecVersion<Runtime>,
      frame_system::CheckTxVersion<Runtime>,
      frame_system::CheckGenesis<Runtime>,
      frame_system::CheckEra<Runtime>,
      frame_system::CheckNonce<Runtime>,
      frame_system::CheckWeight<Runtime>,
      pallet_transaction_payment::ChargeTransactionPayment<Runtime>,
      frame_metadata_hash_extension::CheckMetadataHash<Runtime>,
  );
  ```

  After:

  ```rust
pub type SignedExtra =
cumulus_pallet_weight_reclaim::StorageWeightReclaim<
      Runtime,
      (
          frame_system::CheckNonZeroSender<Runtime>,
          frame_system::CheckSpecVersion<Runtime>,
          frame_system::CheckTxVersion<Runtime>,
          frame_system::CheckGenesis<Runtime>,
          frame_system::CheckEra<Runtime>,
          frame_system::CheckNonce<Runtime>,
          frame_system::CheckWeight<Runtime>,
          pallet_transaction_payment::ChargeTransactionPayment<Runtime>,
          frame_metadata_hash_extension::CheckMetadataHash<Runtime>,
      ),
  >;
  ```

  `from_minimal_extension`:

  Before:

  ```rust
  fn from_minimal_extension(minimal: MinimalExtension) -> Self {
      (
          frame_system::CheckNonZeroSender::<Runtime>::new(),
          frame_system::CheckSpecVersion::<Runtime>::new(),
          frame_system::CheckTxVersion::<Runtime>::new(),
          frame_system::CheckGenesis::<Runtime>::new(),
          frame_system::CheckEra::<Runtime>::from(minimal.era),
          frame_system::CheckNonce::<Runtime>::from(minimal.nonce),
          frame_system::CheckWeight::<Runtime>::new(),

pallet_transaction_payment::ChargeTransactionPayment::<Runtime>::from(minimal.tip),
          frame_metadata_hash_extension::CheckMetadataHash::new(false),
      )
  }
  ```

  After:

  ```rust
  fn from_minimal_extension(minimal: MinimalExtension) -> Self {
      let inner = (
          frame_system::CheckNonZeroSender::<Runtime>::new(),
          frame_system::CheckSpecVersion::<Runtime>::new(),
          frame_system::CheckTxVersion::<Runtime>::new(),
          frame_system::CheckGenesis::<Runtime>::new(),
          frame_system::CheckEra::<Runtime>::from(minimal.era),
          frame_system::CheckNonce::<Runtime>::from(minimal.nonce),
          frame_system::CheckWeight::<Runtime>::new(),

pallet_transaction_payment::ChargeTransactionPayment::<Runtime>::from(minimal.tip),
          frame_metadata_hash_extension::CheckMetadataHash::new(false),
      );
      cumulus_pallet_weight_reclaim::StorageWeightReclaim::new(inner)
  }
  ```

  New pallet config required:

  ```rust
  impl cumulus_pallet_weight_reclaim::Config for Runtime {
      type WeightInfo = ();
  }
  ```

---

- **Short description**

For EVM runtimes using Frontier, both block building and block import
must now register the `ProofSizeExt` host extension. Frontier EVM
(stable2503) calls `get_proof_size()` at multiple points during EVM
execution. If `ProofSizeExt` is present during block building but absent
during block import on syncing nodes, `get_proof_size()` returns
different values on each path, leading to different gas/weight
calculations, different state roots, and digest mismatch panics.

  Two changes are needed in the node service:
1. `ProposerFactory::new` must be replaced with
`ProposerFactory::with_proof_recording`
2. `sc_service::new_full_parts` must be replaced with
`sc_service::new_full_parts_record_import` with proof recording enabled

- **Who is affected**

* **Downstream node implementations** that run Frontier EVM
(solochain-evm or any EVM-enabled runtime)

Non-EVM runtimes (e.g. parachain-only) are not affected by this specific
change, though using `with_proof_recording` is still recommended for
accurate `StorageWeightReclaim` operation.

- **Suggested code changes**

  Block building — ProposerFactory:

  Before:

  ```rust
  let proposer_factory = sc_basic_authorship::ProposerFactory::new(
      task_manager.spawn_handle(),
      client.clone(),
      transaction_pool.clone(),
      prometheus_registry.as_ref(),
      telemetry.as_ref().map(|t| t.handle()),
  );
  ```

  After:

  ```rust
let proposer_factory =
sc_basic_authorship::ProposerFactory::with_proof_recording(
      task_manager.spawn_handle(),
      client.clone(),
      transaction_pool.clone(),
      prometheus_registry.as_ref(),
      telemetry.as_ref().map(|t| t.handle()),
  );
  ```

  Block import — Client initialization:

  Before:

  ```rust
  let (client, backend, keystore_container, task_manager) =
      sc_service::new_full_parts::<Block, RuntimeApi, _>(
          config,
          telemetry.as_ref().map(|(_, telemetry)| telemetry.handle()),
          executor,
      )?;
  ```

  After:

  ```rust
  let (client, backend, keystore_container, task_manager) =
      sc_service::new_full_parts_record_import::<Block, RuntimeApi, _>(
          config,
          telemetry.as_ref().map(|(_, telemetry)| telemetry.handle()),
          executor,
          true, // enable proof recording on import
      )?;
  ```

---

- **Short description**

StorageHub now targets polkadot-sdk `stable2503` and frontier
`stable2503`. Downstream consumers must align their polkadot-sdk and
frontier dependency versions. This brings several inherited breaking
changes from the Polkadot SDK upgrade.

- **Who is affected**

* **All downstream runtimes and node implementations** that depend on
StorageHub crates

- **Suggested code changes**

* **`sp_std` deprecated**
([polkadot-sdk#7043](paritytech/polkadot-sdk#7043)):
Replace all `sp_std::*` imports with `alloc::*` or `core::*`
equivalents. Add `extern crate alloc;` where needed.

    ```rust
    // Before
    use sp_std::{collections::btree_map::BTreeMap, vec::Vec};

    // After
    extern crate alloc;
    use alloc::{collections::BTreeMap, vec::Vec};
    ```

* **Genesis config presets**
([polkadot-sdk#6349](paritytech/polkadot-sdk#6349)):
Migrate from `serde_json::to_value()` to the `build_struct_json_patch!`
macro.

    ```rust
    // Before
    let config = RuntimeGenesisConfig {
        balances: BalancesConfig { ... },
        ..Default::default()
    };
serde_json::to_value(config).expect("Could not build genesis config.")

    // After
    use frame_support::build_struct_json_patch;
    build_struct_json_patch!(RuntimeGenesisConfig {
        balances: BalancesConfig { ... },
    })
    ```

  * **`pallet_session::Config`**: Add `type DisablingStrategy = ();`

* **`DecodeWithMemTracking`**
([polkadot-sdk#7634](paritytech/polkadot-sdk#7634)):
Custom types used at extrinsic boundaries or in `RuntimeCall` must
derive `DecodeWithMemTracking` from the `parity-scale-codec` crate.

* **`litep2p` default network backend**
([polkadot-sdk#8461](paritytech/polkadot-sdk#8461)):
Substrate chains now use `litep2p` instead of `libp2p` by default. While
`litep2p` is wire-compatible with `libp2p`, behavioral differences exist
(e.g. notification protocol handling, error mapping for
request-response).


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/moonsong-labs/storage-hub/pull/671"
target="_blank">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
gonzamontiel pushed a commit to datahaven-xyz/datahaven that referenced this pull request Mar 26, 2026
## Polkadot upgrade 2503

In this PR, we upgrade form Polkadot SDK 2412 to Polkadot SDK 2503. In
order to upgrade the SDK we need to upgrade some dependencies :
StorageHub and frontier simultaneously.


## What changes 

### Trivial changes

* paritytech/polkadot-sdk#7634 -> The new trait
is required in all the pallets using scale encoding.

* paritytech/polkadot-sdk#7043 -> Remove
deprecated `sp-std` and replace with `alloc` or `core`.

* paritytech/polkadot-sdk#6140 -> Accurate
weight reclaim with frame_system::WeightReclaim


### Breaking changes

* paritytech/polkadot-sdk#2072 -> There is a
change in `pallet-referenda`. Now, the tracks are retrieved as a list of
`Track`s. Also, the names of the tracks might have some trailing null
values (`\0`). This means display representation of the tracks' names
must be sanitized.

* paritytech/polkadot-sdk#5723 -> adds the
ability for these pallets to specify their source of the block number.
This is useful when these pallets are migrated from the relay chain to a
parachain and vice versa. (Not entirely sure it is breaking as it is
being marked as backward compatible).

* paritytech/polkadot-sdk#6338 -> Update
Referenda to Support Block Number Provider

### Notable changes

* paritytech/polkadot-sdk#5703 -> Not changes
required in the codebase but impact fastSync mode. Should improve
testing.

* paritytech/polkadot-sdk#5842 -> Removes
`libp2p` types in authority-discovery, and replace them with network
backend agnostic types from `sc-network-types`. The `sc-network`
interface is therefore updated accordingly.

## What changes in Frontier 

* polkadot-evm/frontier#1693 -> Add support for
EIP 7702 which has been enable with Pectra. This EIP add a new field
`AuthorizationList` in Ethereum transaction.

Changes example :

```rust
#[test]
fn validate_transaction_fails_on_filtered_call() {
...
            pallet_evm::Call::<Runtime>::call {
                source: H160::default(),
                target: H160::default(),
                input: Vec::new(),
                value: sp_core::U256::zero(),
                gas_limit: 21000,
                max_fee_per_gas: sp_core::U256::zero(),
                max_priority_fee_per_gas: Some(sp_core::U256::zero()),
                nonce: None,
                access_list: Vec::new(),
                authorization_list: Vec::new(),
            }
            .into(),
```

## ⚠️ Breaking Changes ⚠️

* Upgrade to Stirage hub v0.5.1
* Support for Ethereum latest upgrade (txs now have the
`authoriation_list` for support EIP 7702)

---------

Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

T2-pallets This PR/Issue is related to a particular pallet. T9-cumulus This PR/Issue is related to cumulus.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

8 participants