Polkadot AssetHub archive node is getting flooded with logs

[pmensik]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

We have been running this Polkadot AH archive node for a couple of months and everything worked fine until the node was killed because the machine ran out of disk space. Upon investigation, we found out that the syslog bloated to 300 GB within two hours. The log went as follows

Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/transactions/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/block-announces/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/transactions/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/block-announces/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/transactions/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/block-announces/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/transactions/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/block-announces/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/transactions/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/block-announces/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/transactions/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/block-announces/1")
Jan 12 19:11:41 tatum-assethub-polkadot-rpc1 polkadot[24578]: 2026-01-12 19:11:41 [Parachain] transport service closed protocol=Allocated("/68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f/transactions/1")

The issue has happened twice over the course of last 4 days. Unfortunately, on both occasions we deleted the logs immediately since it's an important customer-facing service so this is all we got. However, the node was operating alright until out of a sudden these started to flood the syslog. The node is accompanied by an instance of sidecar running as a separate systemd service. Here's the node config

ExecStart=/usr/local/bin/polkadot \
        --name "node" \
        --base-path '/var/lib/assethub' \
        --telemetry-url 'wss://telemetry.polkadot.io/submit/ 0' \
        --chain asset-hub-polkadot \
        --database paritydb \
        --state-pruning archive \
        --blocks-pruning archive \
        --port 30333 \
        --no-hardware-benchmarks \
        --no-mdns \
        --prometheus-external \
        --enable-offchain-indexing true \
        --db-cache 8000 \
# RPC settings
        --rpc-external \
        --rpc-cors all \
        --rpc-methods safe \
        --in-peers 15 \
        --out-peers 15 \
        --rpc-max-connections 10000 \
        --rpc-max-request-size 5 \
        --rpc-max-response-size 15 \
        --rpc-max-subscriptions-per-connection 1024 \
        --max-runtime-instances 32 \
        --runtime-cache-size 64 \
        -- \
        --state-pruning 100 \
        --blocks-pruning 100 \

Steps to reproduce

No response

[sandreim]

CC @paritytech/networking

[lexnv]

We have seen these issues in the past around litep2p API misuse:

• registered protocols (reqresp / notifications) are designed to be long lived
• although we have a mechanism of dynamic registering / unregistering, we have not adopted this widely in substrate yet

I'm assuming that a registered protocol has been dropped

Could you provide the version for which this happens?
Are you running extra custom patches / have you modified the code in any way?

It would be helpful if this not just a one-off issue to provide around ~100-200 log lines before the first occurrence of transport service closed. This could help us identify the offending protocol 🙏

[Curu24]

hello, we are running polkadot 1.21.0-1b9a876861a
no custom patches or code modifications
happened again on our 2nd RPC just yesterday - here is ~250 log lines before it started https://pastebin.com/kWubj7z0

[lexnv]

@Curu24 @pmensik Are you running with constrained RAM? (Maybe 32 gib or less?)

It seems that litep2p backend has exited, which might be caused by the OOM.

This shows two issues:

• notification protocols are still active as zombie tasks
• network should bring down the whole node, there's no point in having lingering connections when the core of the network has exited

[lexnv]

The following code path causes the network backend to be dropped silently:

polkadot-sdk/substrate/client/service/src/lib.rs

Lines 208 to 217 in e30ff0d

	notification = imported_blocks_stream.next() => {
	let notification = match notification {
	Some(n) => n,
	// If this stream is shut down, that means the client has shut down, and the
	// most appropriate thing to do for the network future is to shut down too.
	None => {
	debug!("Block import stream has terminated, shutting down the network future.");
	return
	},
	};

A counter-part fix has been added to litep2p to avoid busy loops on zombie tasks:

• notif/fix: Avoid CPU busy loops on litep2p full shutdown litep2p#521

[Curu24]

@lexnv both nodes where the event occurred have 64GB of RAM and there are no restrictions.
we don't see any RAM spikes in our monitoring dashboard and the load is constant around 20%
there is also no OOM killer in the system logs