[BUG] build fails for IT tests

[sam-herman]

What is the bug?

Build fails for IT tests.

121 tests completed, 3 failed

=== Standard output of node `node{::integTest-0}` ===

Failure report:

[HybridQueryExplainIT](https://github.com/opensearch-project/neural-search/classes/org.opensearch.neuralsearch.query.HybridQueryExplainIT.html). [testExplain_whenMultipleSubqueriesAndOneShard_thenSuccessful](https://github.com/opensearch-project/neural-search/classes/org.opensearch.neuralsearch.query.HybridQueryExplainIT.html#testExplain_whenMultipleSubqueriesAndOneShard_thenSuccessful)
[HybridQueryExplainIT](https://github.com/opensearch-project/neural-search/classes/org.opensearch.neuralsearch.query.HybridQueryExplainIT.html). [testExplanationResponseProcessor_whenProcessorIsNotConfigured_thenResponseHasQueryExplanations](https://github.com/opensearch-project/neural-search/classes/org.opensearch.neuralsearch.query.HybridQueryExplainIT.html#testExplanationResponseProcessor_whenProcessorIsNotConfigured_thenResponseHasQueryExplanations)
[HybridQuerySortIT](https://github.com/opensearch-project/neural-search/classes/org.opensearch.neuralsearch.query.HybridQuerySortIT.html). [testExplainAndSort_whenIndexWithMultipleShards_thenSuccessful](https://github.com/opensearch-project/neural-search/classes/org.opensearch.neuralsearch.query.HybridQuerySortIT.html#testExplainAndSort_whenIndexWithMultipleShards_thenSuccessful)

How can one reproduce the bug?

./gradlew build

What is the expected behavior?

Build to work.

What is your host/environment?

MacOS: Sequoia
OpenJDK: 23

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

[martin-gaievski]

@sam-herman thank you for reporting the issue. Can you provide additional inforamtion:

• what is your branch and JDK version
• did you get any "reproduce this with" command with the seed for test
• if that's something you're seeing regularly can you isolate just test runs by running "gradle integTest". This sub-command is executed as part of the "gradle build" anyways, but runs faster.

I tried running "gradle build" on linux and JDK 21, got successful results:

./gradlew build
Starting a Gradle Daemon (subsequent builds will be faster)
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.10.2
  OS Info               : Linux 5.10.234-205.895.amzn2int.x86_64 (amd64)
  JDK Version           : 21 (Azul Zulu JDK)
  JAVA_HOME             : /local/home/user/.sdkman/candidates/java/21.0.4-zulu
  Random Testing Seed   : B44066C9E58EE55F
  In FIPS 140 mode      : false
=======================================

> Task :generatePomFileForNebulaPublication
....
           ^
100 warnings

> Task :compileTestFixturesJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :qa:rolling-upgrade:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.10.2/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 9m 19s
98 actionable tasks: 95 executed, 3 up-to-date

[sam-herman]

@sam-herman thank you for reporting the issue. Can you provide additional inforamtion:
what is your branch and JDK version
did you get any "reproduce this with" command with the seed for test
if that's something you're seeing regularly can you isolate just test runs by running "gradle integTest". This sub-command is executed as part of the "gradle build" anyways, but runs faster.

Hi @martin-gaievski I updated the info in the description

MacOS: Sequoia
JDK: 23

[sam-herman]

@martin-gaievski should we add a MacOS build step here:
https://github.com/opensearch-project/neural-search/blob/main/.github/workflows/CI.yml
similar to this https://github.com/opensearch-project/k-NN/blob/main/.github/workflows/CI.yml#L106 ?

[martin-gaievski]

@martin-gaievski should we add a MacOS build step here: https://github.com/opensearch-project/neural-search/blob/main/.github/workflows/CI.yml similar to this https://github.com/opensearch-project/k-NN/blob/main/.github/workflows/CI.yml#L106 ?

reason being mainly to keep CI light, as neural does not use platform native features, that's different from knn plugin in your example, it uses native engines like faiss

[martin-gaievski]

@sam-herman thank you for reporting the issue. Can you provide additional inforamtion:
what is your branch and JDK version
did you get any "reproduce this with" command with the seed for test
if that's something you're seeing regularly can you isolate just test runs by running "gradle integTest". This sub-command is executed as part of the "gradle build" anyways, but runs faster.

Hi @martin-gaievski I updated the info in the description

MacOS: Sequoia
JDK: 23

just tried on a mac with jdk 23, passing for me. Can you please try to repro again and if it happens again note the test seed that leads to the failure. You can skip other checks by executing gradle integTest instead of build

./gradlew build
Starting a Gradle Daemon (subsequent builds will be faster)
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.10.2
  OS Info               : Mac OS X 15.3.1 (aarch64)
  JDK Version           : 23 (Amazon Corretto JDK)
  JAVA_HOME             : /Users/user/.sdkman/candidates/java/23.0.2-amzn
  Random Testing Seed   : A778A2B4EA6E130D
  In FIPS 140 mode      : false
=======================================

> Task :generatePomFileForNebulaPublication
Maven publication 'nebula' pom metadata warnings (silence with 'suppressPomMetadataWarningsFor(variant)'):
  - Variant testFixturesApiElements:
      - Declares capability org.opensearch:neural-search-test-fixtures:3.0.0.0-alpha1-SNAPSHOT which cannot be mapped to Maven
  - Variant testFixturesRuntimeElements:
      - Declares capability org.opensearch:neural-search-test-fixtures:3.0.0.0-alpha1-SNAPSHOT which cannot be mapped to Maven
These issues indicate information that is lost in the published 'pom' metadata file, which may be an issue if the published library is consumed by an old Gradle version or Apache Maven.
The 'module' metadata file, which is used by Gradle 6+ is not affected.

> Task :compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.10.2/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 10m 23s