[BUG] Assertion fails on startup with remote store enabled

[msfroh]

Describe the bug

If you try to enable remote store on a JVM that has assertions enabled (as one should if they're experimenting and want to double-check invariants), it crashes on startup during the flow for RemoteStoreNodeService#createAndVerifyRepositories() because there's an assertion in BlobStoreRepository regarding the threads allowed to to call the blobContainer method.

Related component

Storage:Remote

To Reproduce

Try to launch OpenSearch from your local Git repo with:

./gradlew run -Dtests.opensearch.node.attr.remote_store.segment.repository=my-repo-1 \
                       -Dtests.opensearch.node.attr.remote_store.translog.repository=my-repo-1 \
                       -Dtests.opensearch.node.attr.remote_store.state.repository=my-repo-1 \
                       -Dtests.opensearch.node.attr.remote_store.repository.my-repo-1.type=fs 
                       -Dtests.opensearch.node.attr.remote_store.repository.my-repo-1.settings.location=/tmp/remote_repository \
                       -Dtests.opensearch.path.repo=/tmp/remote_repository

The node fails on startup with:

»  java.lang.AssertionError: Expected current thread [Thread[#1,main,5,main]] to be the snapshot_deletion or snapshot or generic thread.
»       at org.opensearch.repositories.blobstore.BlobStoreRepository.assertSnapshotOrGenericThread(BlobStoreRepository.java:2980)
»       at org.opensearch.repositories.blobstore.BlobStoreRepository.blobContainer(BlobStoreRepository.java:983)
»       at org.opensearch.repositories.url.URLRepository.blobContainer(URLRepository.java:139)
»       at org.opensearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:3738)
»       at org.opensearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:3720)
»       at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2989)
»       at org.opensearch.node.remotestore.RemoteStoreNodeService.createAndVerifyRepositories(RemoteStoreNodeService.java:156)
»       at org.opensearch.node.Node$LocalNodeFactory.apply(Node.java:2221)
»       at org.opensearch.node.Node$LocalNodeFactory.apply(Node.java:2200)
»       at org.opensearch.transport.TransportService.doStart(TransportService.java:313)
»       at org.opensearch.common.lifecycle.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:77)
»       at org.opensearch.node.Node.start(Node.java:1741)
»       at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:346)
»       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:420)
»       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168)
»       at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159)
»       at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110)
»       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
»       at org.opensearch.cli.Command.main(Command.java:101)
»       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125)
»       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91)

Expected behavior

The node should start up.

I was able to hack it by adding || Thread.currentThread().getName().equals("main") as a condition in

OpenSearch/server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

Lines 2976 to 2982 in d990978

	protected void assertSnapshotOrGenericThread() {
	assert Thread.currentThread().getName().contains('[' + ThreadPool.Names.SNAPSHOT_DELETION + ']')
	|| Thread.currentThread().getName().contains('[' + ThreadPool.Names.SNAPSHOT + ']')
	|| Thread.currentThread().getName().contains('[' + ThreadPool.Names.GENERIC + ']') : "Expected current thread ["
	+ Thread.currentThread()
	+ "] to be the snapshot_deletion or snapshot or generic thread.";
	}

Additional Details

N/A