Statement-store: defer statement protocol connections during major sync by DenzelPenzel · Pull Request #11487 · paritytech/polkadot-sdk

DenzelPenzel · 2026-03-24T22:01:29Z

Description

Implement #11411

Reconnect all statement protocol peers when major sync ends. Removing and re-adding peers to the reserved set closes and reopens the notification substreams on both sides, triggering a fresh bidirectional initial sync. This causes peers to re-deliver any statements that were dropped during the sync window.

On was_major_syncing → false transition, call reconnect_statement_peers which removes then re-adds all connected peers from the statement
protocol reserved set
- Both sides receive NotificationStreamClosed then NotificationStreamOpened, restarting the 100ms initial sync burst

Changes

reconnect_statement_peers: fires when major sync ends, reconnects all statement peers to trigger fresh initial sync
New zombienet integration test: statement_store_peer_disconnect_during_major_sync — verifies that statements submitted before a late-joining nodeenters major sync are fully recovered after sync completes, confirmed via log assertion that reconnect_statement_peers fired

DenzelPenzel · 2026-03-27T14:52:10Z

/cmd prdoc --audience node_dev --bump patch

…e_dev --bump patch'

s0me0ne-unkn0wn

Looks good (modulo my poor knowledge of Substrate networking internals)

…ajor-sync

…heck

…ent-store-loss-during-major-sync # Conflicts: # cumulus/zombienet/zombienet-sdk/tests/zombie_ci/statement_store.rs # cumulus/zombienet/zombienet-sdk/tests/zombie_ci/statement_store/bench.rs

DenzelPenzel · 2026-04-02T10:34:45Z

/cmd fmt

AndreiEres

Overall ok, but feel uneasy that we loose the peerset on error in reconnect_statement_peers. But zombienet test doesn't catch if the fix works

… github.com:paritytech/polkadot-sdk into denzelpenzel/statement-store-loss-during-major-sync # Conflicts: # cumulus/zombienet/zombienet-sdk/tests/zombie_ci/statement_store/integration.rs

DenzelPenzel · 2026-04-13T12:57:16Z

/cmd fmt

AndreiEres

LGTM

alexggh · 2026-04-14T14:10:40Z

+/// Scenario:
+/// 1. Spawn charlie only and let the relay chain advance ~10 blocks
+/// 2. Submit multiple statements to charlie
+/// 3. Add dave as a late joiner — dave will enter major sync because the chain has already


Do you have anything to proove 3 happens ?

Yes, this cond confirm it

polkadot-sdk/cumulus/zombienet/zombienet-sdk/tests/zombie_ci/statement_store/integration.rs

Line 495 in fc46117

dave_logs.lines().any(|l| l.contains("Major sync complete, reconnecting")),

But ... this confirms just that you had a major sync happening not that, you received statements while you were in major sync.

bkchr

The current implementation is trying to achieve the goal in a very hacky way. We should just not add the peers to the reserved set as long as we are in major sync mode.

bkchr · 2026-04-15T08:39:55Z

 			SyncEvent::PeerConnected(remote) => {
 				let addr = iter::once(multiaddr::Protocol::P2p(remote.into()))
 					.collect::<multiaddr::Multiaddr>();
 				let result = self.network.add_peers_to_reserved_set(


You should not do this when you are in major sync mode. There is no need to connect to the other side, as long as you are doing the major sync.

When the major sync is then ready, you can start adding them to the reserved set and the substream should be opened.

Agreed that adding new peers to the reserved set during major sync is the right approach for late joiners we can buffer them and add them cleanly when sync ends, avoiding any substream open during the sync window.

However, there is a gap for already-connected peers (nodes whose substream was open before major sync started) we don't have a way to re-trigger it without forcing a close+reopen. So even with the reserved-set approach we are still need something equivalent to the remove+add for already-connected peers to recover missed statements, wdty? @bkchr

However, there is a gap for already-connected peers (nodes whose substream was open before major sync started)

Major sync should start immediately, the moment the first peer is connected.

Looking at this:

polkadot-sdk/substrate/client/network/sync/src/strategy/chain_sync.rs

Line 944 in d7bc514

if target > best_block && target - best_block > MAJOR_SYNC_BLOCKS.into() {

There still seems to be a gap where if fall behind 5 blocks, we get into major sync and the we will drop statements here, from nodes that are already connected:

https://github.com/paritytech/polkadot-sdk/pull/11487/changes#diff-1b228aee91454a7222489dbb0382d7a7161a19f17048cb2de2d7845f5cac533aL925.

So maybe the mitigation for this situations is to also accept statements when we are in major sync, I think that should be good enough.

Or if we want to be bullet proof we could note we dropped statements because it found us in a major sync and have a timer every(1 minute) where we remove some random peer from reserved set if we dropped statements(forcing a disconnect basically) and then next minute we re-add it to reserved set. I would just disconnect a few peers because we don't want to re-sync with all of them.

… reconnec

…ent-store-loss-during-major-sync # Conflicts: # substrate/client/network/statement/src/lib.rs

…s-during-major-sync' into denzelpenzel/statement-store-loss-during-major-sync

DenzelPenzel · 2026-04-16T21:54:45Z

/cmd fmt

statement-store: defer statement protocol connections during major sync

7d0c13d

DenzelPenzel requested a review from alexggh March 24, 2026 22:01

alexggh reviewed Mar 25, 2026

View reviewed changes

Comment thread substrate/client/network/statement/src/lib.rs

DenzelPenzel marked this pull request as draft March 25, 2026 10:14

DenzelPenzel marked this pull request as ready for review March 26, 2026 12:10

statement-store: major sync tests

c8a9ed7

DenzelPenzel requested review from alexggh, bkchr and s0me0ne-unkn0wn March 26, 2026 12:48

Update from github-actions[bot] running command 'fmt'

a382525

DenzelPenzel added the T0-node This PR/Issue is related to the topic “node”. label Mar 27, 2026

DenzelPenzel added 2 commits March 27, 2026 09:44

statement-store: clippy

7cedec4

statement-store: fix spawn_network to support single-collator tests,

5a8e3f1

Update from github-actions[bot] running command 'prdoc --audience nod…

b87cd17

…e_dev --bump patch'

s0me0ne-unkn0wn approved these changes Mar 31, 2026

View reviewed changes

Comment thread substrate/client/network/statement/src/lib.rs Outdated

DenzelPenzel added 2 commits March 31, 2026 15:47

statement-store: use BTreeMap for deferred_peers

8db0b1a

statement-store: clippy

2120c5d

DenzelPenzel changed the title ~~statement-store: defer statement protocol connections during major sync~~ Statement-store: defer statement protocol connections during major sync Mar 31, 2026

Merge branch 'master' into denzelpenzel/statement-store-loss-during-m…

038e8e5

…ajor-sync

alexggh reviewed Apr 1, 2026

View reviewed changes

Comment thread substrate/client/network/statement/src/lib.rs Outdated

alexggh reviewed Apr 1, 2026

View reviewed changes

Comment thread substrate/client/network/statement/src/lib.rs

alexggh reviewed Apr 1, 2026

View reviewed changes

Comment thread substrate/client/network/statement/src/lib.rs

alexggh reviewed Apr 1, 2026

View reviewed changes

Comment thread cumulus/zombienet/zombienet-sdk/tests/zombie_ci/statement_store.rs Outdated

statement-store: fix deferred peer race and extract sync transition c…

0576365

…heck

DenzelPenzel requested a review from alexggh April 1, 2026 11:26

DenzelPenzel added 2 commits April 2, 2026 09:47

Merge remote-tracking branch 'origin/master' into denzelpenzel/statem…

b0a6db0

…ent-store-loss-during-major-sync # Conflicts: # cumulus/zombienet/zombienet-sdk/tests/zombie_ci/statement_store.rs # cumulus/zombienet/zombienet-sdk/tests/zombie_ci/statement_store/bench.rs

statement-store: address review feedback

2934407

DenzelPenzel force-pushed the denzelpenzel/statement-store-loss-during-major-sync branch from 555918e to 2934407 Compare April 2, 2026 10:26

DenzelPenzel requested review from AndreiEres and alexggh April 10, 2026 15:01

AndreiEres reviewed Apr 10, 2026

View reviewed changes

DenzelPenzel and others added 4 commits April 13, 2026 12:38

statement-store: recover statements lost during major sync

79890d7

statement-store: check statements

e2118dd

Update from github-actions[bot] running command 'fmt'

a8d7939

statement-store: incr timeout

b69e515

DenzelPenzel force-pushed the denzelpenzel/statement-store-loss-during-major-sync branch from 0c0b061 to 66157b3 Compare April 13, 2026 12:40

Merge branch 'denzelpenzel/statement-store-loss-during-major-sync' of…

b076543

… github.com:paritytech/polkadot-sdk into denzelpenzel/statement-store-loss-during-major-sync # Conflicts: # cumulus/zombienet/zombienet-sdk/tests/zombie_ci/statement_store/integration.rs

DenzelPenzel force-pushed the denzelpenzel/statement-store-loss-during-major-sync branch from 66157b3 to b076543 Compare April 13, 2026 12:51

Update from github-actions[bot] running command 'fmt'

fc46117

DenzelPenzel requested a review from AndreiEres April 13, 2026 18:33

AndreiEres reviewed Apr 14, 2026

View reviewed changes

AndreiEres approved these changes Apr 14, 2026

View reviewed changes

alexggh requested changes Apr 14, 2026

View reviewed changes

statement-store: improve naming

2af5257

bkchr requested changes Apr 15, 2026

View reviewed changes

DenzelPenzel added 3 commits April 16, 2026 11:46

statement-store: buffer peers during major sync instead of remove+add…

c96de87

… reconnec

Merge remote-tracking branch 'origin/master' into denzelpenzel/statem…

922309e

…ent-store-loss-during-major-sync # Conflicts: # substrate/client/network/statement/src/lib.rs

statement-store: unit test with_syncing

637f1b0

DenzelPenzel force-pushed the denzelpenzel/statement-store-loss-during-major-sync branch from f6f1aa8 to 894595d Compare April 16, 2026 11:42

statement-store: update prdoc

1803dab

DenzelPenzel force-pushed the denzelpenzel/statement-store-loss-during-major-sync branch from 894595d to 1803dab Compare April 16, 2026 11:44

github-actions bot and others added 4 commits April 16, 2026 11:49

Update from github-actions[bot] running command 'fmt'

e09bd4c

statement-store: recover statements dropped during major sync

0a86fbf

Merge remote-tracking branch 'origin/denzelpenzel/statement-store-los…

76f2487

…s-during-major-sync' into denzelpenzel/statement-store-loss-during-major-sync

statement-store: improve recovery integration test coverage

6e86175

Update from github-actions[bot] running command 'fmt'

6a90c6b

Conversation

DenzelPenzel commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Uh oh!

Uh oh!

DenzelPenzel commented Mar 27, 2026

Uh oh!

s0me0ne-unkn0wn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DenzelPenzel commented Apr 2, 2026

Uh oh!

AndreiEres left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DenzelPenzel commented Apr 13, 2026

Uh oh!

AndreiEres left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexggh Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

DenzelPenzel Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

alexggh Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bkchr left a comment

Choose a reason for hiding this comment

Uh oh!

bkchr Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

DenzelPenzel Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bkchr Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

alexggh Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

DenzelPenzel commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

DenzelPenzel commented Mar 24, 2026 •

edited

Loading

DenzelPenzel Apr 15, 2026 •

edited

Loading