[AUTOCUT] Gradle Check Flaky Test Report for SimpleSearchIT

[opensearch-ci-bot]

Flaky Test Report for SimpleSearchIT

Noticed the SimpleSearchIT has some flaky, failing tests that failed during post-merge actions.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
069c2d8	18097	57263	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
0d1bb9b	18035	56909	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
15b702d	18042	56934	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
18a3b75	17796	56046	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
3a7d8fa	17976	56634	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
436038d	18041	56989	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
59a6efd	18036	57036	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
8cc58b5	18058	57023	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
967eee1	17216	56041	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
b842e48	17902	56242	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
c44d230	17749	56412	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
e61b6ab	18005	56703	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
e7ed33f	17978	57115	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}} org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
693c788	17921	56337	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
caf5d71	18039	56924	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
d18982c	17912	56693	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"true"}}
d29e95c	17882	56063	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}}
d37cc9b	16831	51449	org.opensearch.search.simple.SimpleSearchIT.classMethod
eba12fa	17370	56995	org.opensearch.search.simple.SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize {p0={"search.concurrent_segment_search.enabled":"false"}}

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the SimpleSearchIT class are:

• 17939
• 17782
• 17896
• 17950
• 17855
• 17868
• 18006
• 17718
• 17769
• 17843
• 17888
• 17984
• 18054
• 18243
• 17447
• 17532
• 17880
• 17893
• 17336
• 17682
• 17711
• 17759
• 17774
• 17791
• 17884
• 17887
• 17894
• 17923
• 17942
• 17944
• 17951
• 17954
• 17977
• 17996
• 18003
• 18013
• 18017
• 18043
• 18050
• 18059
• 18063
• 18066
• 18069
• 18105
• 18107
• 18189
• 18236
• 13172
• 14912
• 18034
• 18038
• 18094

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[sandeshkr419]

Looking into build logs:

[2024-12-16T08:20:25,037][ERROR][o.o.i.s.IndexShard       ] [node_t0] [test][0] Error Fetching SegmentInfos and latest checkpoint
org.apache.lucene.store.AlreadyClosedException: engine is closed
	at org.opensearch.index.shard.IndexShard.getEngine(IndexShard.java:3728) ~[main/:?]
	at org.opensearch.index.shard.IndexShard.getSegmentInfosSnapshot(IndexShard.java:5367) ~[main/:?]
	at org.opensearch.index.shard.IndexShard.getLatestSegmentInfosAndCheckpoint(IndexShard.java:1728) [main/:?]
	at org.opensearch.index.shard.IndexShard.updateReplicationCheckpoint(IndexShard.java:4868) [main/:?]
	at org.opensearch.index.shard.IndexShard$ReplicationCheckpointUpdater.afterRefresh(IndexShard.java:4862) [main/:?]
	at org.apache.lucene.search.ReferenceManager.notifyRefreshListenersRefreshed(ReferenceManager.java:275) [lucene-core-9.12.0.jar:9.12.0 e913796758de3d9b9440669384b29bec07e6a5cd - 2024-09-25 16:37:02]
	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:182) [lucene-core-9.12.0.jar:9.12.0 e913796758de3d9b9440669384b29bec07e6a5cd - 2024-09-25 16:37:02]
	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) [lucene-core-9.12.0.jar:9.12.0 e913796758de3d9b9440669384b29bec07e6a5cd - 2024-09-25 16:37:02]
	at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1771) [main/:?]
	at org.opensearch.index.engine.InternalEngine.flush(InternalEngine.java:1886) [main/:?]
	at org.opensearch.index.engine.Engine.flush(Engine.java:1214) [main/:?]
	at org.opensearch.index.engine.Engine.flushAndClose(Engine.java:2002) [main/:?]
	at org.opensearch.index.shard.IndexShard.close(IndexShard.java:2056) [main/:?]
	at org.opensearch.index.IndexService.closeShard(IndexService.java:805) [main/:?]
	at org.opensearch.index.IndexService.removeShard(IndexService.java:781) [main/:?]
	at org.opensearch.index.IndexService.close(IndexService.java:495) [main/:?]
	at org.opensearch.indices.IndicesService.removeIndex(IndicesService.java:1160) [main/:?]
	at org.opensearch.indices.cluster.IndicesClusterStateService.removeIndices(IndicesClusterStateService.java:443) [main/:?]
	at org.opensearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:284) [main/:?]
	at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:637) [main/:?]
	at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:623) [main/:?]
	at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:589) [main/:?]
	at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:510) [main/:?]
	at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:205) [main/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:932) [main/:?]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283) [main/:?]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246) [main/:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1575) [?:?]
[2024-12-16T08:20:25,057][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode 

From initial look, it looks more like OpenSearch process failure or corrupted index failure.

[bowenlan-amzn]

SimpleSearchIT.testSimpleTerminateAfterTrackTotalHitsUpToSize is flaky

[prudhvigodithi]

I was able to re-produce this with a seed

./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.simple.SimpleSearchIT"  -Dtests.seed=7560B20A7AC502D

@getsaurabh02 @andrross

[andrross]

This is another case that looks like it was introduced by #17769. FYI @harshavamsi