besu OOM when no peers, no syncing

[macfarla]

Steps to Reproduce

start besu with --discovery-enabled=false to force zero peers
come back next day

this image is SNAP server-enabled (green) node syncing. Client has zero peers, intention was to add server as a peer once it's synced (testing SNAP server)

running this version off a branch but I'm guessing it's a pre-existing issue
besu/v26.3-develop-744a9e9/linux-aarch_64/openjdk-java-21

Expected behavior: [What you expect to happen]
besu continues to run, and is able to resume syncing when peers are available

Actual behavior: [What actually happens]
OOM after some time

sallymacfarlane@dev-elc-bu-nb-hoodi-snap-rate-limit-client-test-11:~$ tail /var/log/besu/besu.log
{"@timestamp":"2026-03-05T12:23:47,519","level":"INFO","thread":"EthScheduler-Timer-0","class":"SnapSyncChainDownloadPipelineFactory","message":"Creating backward header download pipeline from pivot=2356711 down to lowest block=0, parallelism=8, batchSize=192","throwable":""}
{"@timestamp":"2026-03-05T12:28:10,837","level":"WARN","thread":"EthScheduler-Services-203 (downloadBackwardHeaders)","class":"SnapSyncChainDownloader","message":"Chain sync encountered error, will retry from saved state","throwable":"java.util.concurrent.CompletionException: org.hyperledger.besu.services.pipeline.exception.AsyncOperationException: Async operation failed. java.util.concurrent.TimeoutException\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)\n\tat java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)\n\tat org.hyperledger.besu.services.pipeline.Pipeline.abort(Pipeline.java:220)\n\tat org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:199)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: org.hyperledger.besu.services.pipeline.exception.AsyncOperationException: Async operation failed. java.util.concurrent.TimeoutException\n\tat org.hyperledger.besu.services.pipeline.AsyncOperationProcessor.outputNextCompletedTask(AsyncOperationProcessor.java:92)\n\tat org.hyperledger.besu.services.pipeline.AsyncOperationProcessor.processNextInput(AsyncOperationProcessor.java:66)\n\tat org.hyperledger.besu.services.pipeline.ProcessingStage.run(ProcessingStage.java:38)\n\tat org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:183)\n\t... 5 more\nCaused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException\n\tat java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)\n\tat java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)\n\tat org.hyperledger.besu.services.pipeline.AsyncOperationProcessor.waitForAnyFutureToComplete(AsyncOperationProcessor.java:100)\n\tat org.hyperledger.besu.services.pipeline.AsyncOperationProcessor.outputNextCompletedTask(AsyncOperationProcessor.java:83)\n\t... 8 more\nCaused by: java.util.concurrent.TimeoutException\n\tat java.base/java.util.concurrent.CompletableFuture$Timeout.run(CompletableFuture.java:2920)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\t... 3 more\n"}
{"@timestamp":"2026-03-05T12:37:30,974","level":"INFO","thread":"EthScheduler-Timer-0","class":"SnapSyncChainDownloadPipelineFactory","message":"Creating backward header download pipeline from pivot=2356711 down to lowest block=0, parallelism=8, batchSize=192","throwable":""}
{"@timestamp":"2026-03-05T13:17:54,145","level":"ERROR","thread":"EthScheduler-Services-282","class":"Besu","message":"Uncaught exception in thread "EthScheduler-Services-282"","throwable":"java.lang.OutOfMemoryError: Java heap space\n"}
{"@timestamp":"2026-03-05T13:40:35,430","level":"ERROR","thread":"EthScheduler-Services-277","class":"Besu","message":"Uncaught exception in thread "EthScheduler-Services-277"","throwable":"java.lang.OutOfMemoryError: Java heap space\n"}
{"@timestamp":"2026-03-05T15:04:49,874","level":"ERROR","thread":"EthScheduler-Services-386","class":"Besu","message":"Uncaught exception in thread "EthScheduler-Services-386"","throwable":"java.lang.OutOfMemoryError: Java heap space\n"}
{"@timestamp":"2026-03-05T15:04:49,874","level":"ERROR","thread":"EthScheduler-Services-388","class":"Besu","message":"Uncaught exception in thread "EthScheduler-Services-388"","throwable":"java.lang.OutOfMemoryError: Java heap space\n"}
{"@timestamp":"2026-03-05T15:37:45,909","level":"ERROR","thread":"EthScheduler-Services-438","class":"Besu","message":"Uncaught exception in thread "EthScheduler-Services-438"","throwable":"java.lang.OutOfMemoryError: Java heap space\n"}
{"@timestamp":"2026-03-05T16:46:36,785","level":"ERROR","thread":"EthScheduler-Services-544","class":"Besu","message":"Uncaught exception in thread "EthScheduler-Services-544"","throwable":"java.lang.OutOfMemoryError: Java heap space\n"}
{"@timestamp":"2026-03-05T17:48:18,850","level":"ERROR","thread":"EthScheduler-Services-615","class":"Besu","message":"Uncaught exception in thread "EthScheduler-Services-615"","throwable":"java.lang.OutOfMemoryError: Java heap space\n"}

Frequency: [What percentage of the time does it occur?]
Moderate to rare?
But could indicate a mem leak that is present regardless

[macfarla]

debug logs

{"@timestamp":"2026-03-05T19:06:30,841","level":"DEBUG","thread":"EthScheduler-Services-70","class":"GetHeadersFromPeerTask","message":"Constructing GetHeadersFromPeerTask with hash null, number 2343078, maxHeaders 192, skip 0, direction REVERSE","throwable":""}
{"@timestamp":"2026-03-05T19:06:30,841","level":"DEBUG","thread":"EthScheduler-Services-119","class":"GetHeadersFromPeerTask","message":"Constructing GetHeadersFromPeerTask with hash null, number 2355750, maxHeaders 192, skip 0, direction REVERSE","throwable":""}
{"@timestamp":"2026-03-05T19:06:30,841","level":"DEBUG","thread":"EthScheduler-Services-70","class":"GetHeadersFromPeerTask","message":"Constructing GetHeadersFromPeerTask with hash null, number 2330598, maxHeaders 192, skip 0, direction REVERSE","throwable":""}
{"@timestamp":"2026-03-05T19:06:30,841","level":"DEBUG","thread":"EthScheduler-Services-101","class":"GetHeadersFromPeerTask","message":"Constructing GetHeadersFromPeerTask with hash null, number 2326566, maxHeaders 192, skip 0, direction REVERSE","throwable":""}
{"@timestamp":"2026-03-05T19:06:30,841","level":"DEBUG","thread":"EthScheduler-Services-113","class":"GetHeadersFromPeerTask","message":"Constructing GetHeadersFromPeerTask with hash null, number 2326182, maxHeaders 192, skip 0, direction REVERSE","throwable":""}
{"@timestamp":"2026-03-05T19:06:30,841","level":"DEBUG","thread":"EthScheduler-Services-84","class":"GetHeadersFromPeerTask","message":"Constructing GetHeadersFromPeerTask with hash null, number 2347110, maxHeaders 192, skip 0, direction REVERSE","throwable":""}
{"@timestamp":"2026-03-05T19:06:30,841","level":"DEBUG","thread":"EthScheduler-Services-76","class":"GetHeadersFromPeerTask","message":"Constructing GetHeadersFromPeerTask with hash null, number 2335398, maxHeaders 192, skip 0, direction REVERSE","throwable":""}
{"@timestamp":"2026-03-05T19:06:30,841","level":"DEBUG","thread":"EthScheduler-Services-98","class":"GetHeadersFromPeerTask","message":"Constructing GetHeadersFromPeerTask with hash null, number 2346534, maxHeaders 192, skip 0, direction REVERSE","throwable":""}
{"@timestamp":"2026-03-05T19:06:30,841","level":"DEBUG","thread":"EthScheduler-Services-70","class":"GetHeadersFromPeerTask","message":"Constructing GetHeadersFromPeerTask with hash null, number 2327334, maxHeaders 192, skip 0, direction REVERSE","throwable":""}

[macfarla]

dl-delay-if-no-peers 1,2,3 are built off commit
besu/v26.3-develop-e6d1cec/linux-aarch_64/openjdk-java-21
delay-if-no-peers
main...macfarla:besu:delay-if-no-peers

dl-delay-if-no-peers-and-cancel-2 is running
besu/v26.3-develop-edd259e/linux-aarch_64/openjdk-java-21
which is that delay plus handling cancellation exception
main...macfarla:besu:delay-if-no-peers-and-handle-cancellation-ex

both versions look better. screenshot shows about 6hrs. in the problem scenario, at 6hrs it was crossing 1GB and here they are all still under 450MB