Merge branch 'main' into feature/unsigned-long-star-tree

Signed-off-by: Shailesh Singh <[email protected]>

Author: Shailesh-Kumar-Singh

CHANGELOG.md

@@ -21,13 +21,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - [Security Manager Replacement] Add a policy parser for Java agent security policies ([#17753](https://github.com/opensearch-project/OpenSearch/pull/17753))
 - [Security Manager Replacement] Implement File Interceptor and add integration tests ([#17760](https://github.com/opensearch-project/OpenSearch/pull/17760))
 - [Security Manager Replacement] Enhance Java Agent to intercept Runtime::halt ([#17757](https://github.com/opensearch-project/OpenSearch/pull/17757))
+- [Security Manager Replacement] Phase off SecurityManager usage in favor of Java Agent ([#17861](https://github.com/opensearch-project/OpenSearch/pull/17861))
 - Support AutoExpand for SearchReplica ([#17741](https://github.com/opensearch-project/OpenSearch/pull/17741))
 - Implement fixed interval refresh task scheduling ([#17777](https://github.com/opensearch-project/OpenSearch/pull/17777))
+- [Tiered caching] Create a single cache manager for all the disk caches. ([#17513](https://github.com/opensearch-project/OpenSearch/pull/17513))
 - Add GRPC DocumentService and Bulk endpoint ([#17727](https://github.com/opensearch-project/OpenSearch/pull/17727))
 - Added scale to zero (`search_only` mode) support for OpenSearch reader writer separation ([#17299](https://github.com/opensearch-project/OpenSearch/pull/17299)
 - [Star Tree] [Search] Resolving numeric range aggregation with metric aggregation using star-tree ([#17273](https://github.com/opensearch-project/OpenSearch/pull/17273))
 - Added Search Only strict routing setting ([#17803](https://github.com/opensearch-project/OpenSearch/pull/17803))
 - Disable the index API for ingestion engine ([#17768](https://github.com/opensearch-project/OpenSearch/pull/17768))
+- Add SearchService and Search GRPC endpoint ([#17830](https://github.com/opensearch-project/OpenSearch/pull/17830))
+- Add update and delete support in pull-based ingestion ([#17822](https://github.com/opensearch-project/OpenSearch/pull/17822))
+- Allow maxPollSize and pollTimeout in IngestionSource to be configurable ([#17863](https://github.com/opensearch-project/OpenSearch/pull/17863))
 - [Star Tree] [Search] Add query changes to support unsigned-long in star tree ([#17275](https://github.com/opensearch-project/OpenSearch/pull/17275))
 
 ### Changed
@@ -36,6 +41,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Unwrap singleton DocValues in global ordinal value source of composite histogram aggregation ([#17740](https://github.com/opensearch-project/OpenSearch/pull/17740))
 - Unwrap singleton DocValues in date histogram aggregation. ([#17643](https://github.com/opensearch-project/OpenSearch/pull/17643))
 - Introduce 512 byte limit to search and ingest pipeline IDs ([#17786](https://github.com/opensearch-project/OpenSearch/pull/17786))
+- Avoid skewed segment replication lag metric ([#17831](https://github.com/opensearch-project/OpenSearch/pull/17831))
 
 ### Dependencies
 - Bump `com.nimbusds:nimbus-jose-jwt` from 9.41.1 to 10.0.2 ([#17607](https://github.com/opensearch-project/OpenSearch/pull/17607), [#17669](https://github.com/opensearch-project/OpenSearch/pull/17669))
@@ -63,6 +69,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ### Removed
 - Remove deprecated `batch_size` parameter from `_bulk` ([#14283](https://github.com/opensearch-project/OpenSearch/issues/14283))
+- Remove `FeatureFlags.APPROXIMATE_POINT_RANGE_QUERY_SETTING` since range query approximation is no longer experimental ([#17769](https://github.com/opensearch-project/OpenSearch/pull/17769))
 
 ### Fixed
 - Fix bytes parameter on `_cat/recovery` ([#17598](https://github.com/opensearch-project/OpenSearch/pull/17598))

build.gradle

@@ -433,11 +433,12 @@ gradle.projectsEvaluated {
 
     project.tasks.withType(Test) { task ->
       if (task != null) {
-        if (BuildParams.runtimeJavaVersion > JavaVersion.VERSION_17) {
-          task.jvmArgs += ["-Djava.security.manager=allow"]
-        }
-        if (BuildParams.runtimeJavaVersion >= JavaVersion.VERSION_20) {
-          task.jvmArgs += ["--add-modules=jdk.incubator.vector"]
+        task.jvmArgs += ["--add-modules=jdk.incubator.vector"]
+
+        // Add Java Agent for security sandboxing
+        if (!(project.path in [':build-tools', ":libs:agent-sm:bootstrap", ":libs:agent-sm:agent"])) {
+          dependsOn(project(':libs:agent-sm:agent').prepareAgent)
+          jvmArgs += ["-javaagent:" + project(':libs:agent-sm:agent').jar.archiveFile.get()]
         }
       }
     }

buildSrc/build.gradle

@@ -110,12 +110,12 @@ dependencies {
   api 'com.netflix.nebula:gradle-info-plugin:12.1.6'
   api 'org.apache.rat:apache-rat:0.15'
   api "commons-io:commons-io:${props.getProperty('commonsio')}"
-  api "net.java.dev.jna:jna:5.14.0"
+  api "net.java.dev.jna:jna:5.16.0"
   api 'com.gradleup.shadow:shadow-gradle-plugin:8.3.5'
   api 'org.jdom:jdom2:2.0.6.1'
   api "org.jetbrains.kotlin:kotlin-stdlib-jdk8:${props.getProperty('kotlin')}"
   api 'de.thetaphi:forbiddenapis:3.8'
-  api 'com.avast.gradle:gradle-docker-compose-plugin:0.17.6'
+  api 'com.avast.gradle:gradle-docker-compose-plugin:0.17.12'
   api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
   api 'org.apache.maven:maven-model:3.9.6'
   api 'com.networknt:json-schema-validator:1.2.0'

buildSrc/src/main/java/org/opensearch/gradle/OpenSearchTestBasePlugin.java

@@ -115,9 +115,6 @@ public void execute(Task t) {
                             test.jvmArgs("--illegal-access=warn");
                         }
                     }
-                    if (test.getJavaVersion().compareTo(JavaVersion.VERSION_17) > 0) {
-                        test.jvmArgs("-Djava.security.manager=allow");
-                    }
                 }
             });
             test.getJvmArgumentProviders().add(nonInputProperties);

client/rest-high-level/src/test/resources/org/opensearch/bootstrap/test.policy

@@ -8,4 +8,5 @@
 
 grant {
   permission java.net.SocketPermission "*", "connect,resolve";
+  permission java.net.NetPermission "accessUnixDomainSocket";
 };

distribution/archives/build.gradle

@@ -38,6 +38,9 @@ CopySpec archiveFiles(CopySpec modulesFiles, String distributionType, String pla
       into('lib') {
         with libFiles()
       }
+      into('agent') {
+        with agentFiles()
+      }
       into('config') {
         dirPermissions {
           unix 0750
@@ -226,3 +229,9 @@ subprojects {
 
   group = "org.opensearch.distribution"
 }
+
+tasks.each {
+  if (it.name.startsWith("build")) {
+    it.dependsOn project(':libs:agent-sm:agent').assemble
+  }
+}

distribution/build.gradle

@@ -357,6 +357,18 @@ configure(subprojects.findAll { ['archives', 'packages'].contains(it.name) }) {
       }
     }
 
+    agentFiles = {
+      copySpec {
+        from(project(':libs:agent-sm:agent').prepareAgent) {
+          include '**/*.jar'
+          exclude '**/*-javadoc.jar'
+          exclude '**/*-sources.jar'
+          // strip the version since jvm.options is using agent without version
+          rename("opensearch-agent-${project.version}.jar", "opensearch-agent.jar")
+        }
+      }
+    }
+
     modulesFiles = { platform ->
       copySpec {
         eachFile {

distribution/src/config/jvm.options

@@ -76,16 +76,12 @@ ${error.file}
 # JDK 9+ GC logging
 9-:-Xlog:gc*,gc+age=trace,safepoint:file=${loggc}:utctime,pid,tags:filecount=32,filesize=64m
 
-# Explicitly allow security manager (https://bugs.openjdk.java.net/browse/JDK-8270380)
-18-:-Djava.security.manager=allow
-
 # JDK 20+ Incubating Vector Module for SIMD optimizations;
 # disabling may reduce performance on vector optimized lucene
 20-:--add-modules=jdk.incubator.vector
 
-# HDFS ForkJoinPool.common() support by SecurityManager
--Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory
-
 # See please https://bugs.openjdk.org/browse/JDK-8341127 (openjdk/jdk#21283)
 23:-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.setAsTypeCache
 23:-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.asTypeUncached
+
+21-:-javaagent:agent/opensearch-agent.jar

distribution/tools/launchers/src/main/java/org/opensearch/tools/launchers/SystemJvmOptions.java

@@ -77,21 +77,11 @@ static List<String> systemJvmOptions() {
                 // log4j 2
                 "-Dlog4j.shutdownHookEnabled=false",
                 "-Dlog4j2.disable.jmx=true",
-                // security manager
-                allowSecurityManagerOption(),
                 javaLocaleProviders()
             )
         ).stream().filter(e -> e.isEmpty() == false).collect(Collectors.toList());
     }
 
-    private static String allowSecurityManagerOption() {
-        if (Runtime.version().feature() > 17) {
-            return "-Djava.security.manager=allow";
-        } else {
-            return "";
-        }
-    }
-
     private static String maybeShowCodeDetailsInExceptionMessages() {
         if (Runtime.version().feature() >= 14) {
             return "-XX:+ShowCodeDetailsInExceptionMessages";

gradle/ide.gradle

@@ -82,9 +82,7 @@ if (System.getProperty('idea.active') == 'true') {
         runConfigurations {
           defaults(JUnit) {
             vmParameters = '-ea -Djava.locale.providers=SPI,CLDR'
-            if (BuildParams.runtimeJavaVersion > JavaVersion.VERSION_17) {
-              vmParameters += ' -Djava.security.manager=allow'
-            }
+            vmParameters += ' -javaagent:' + project(':libs:agent-sm:agent').jar.archiveFile.get()
           }
         }
         copyright {

libs/agent-sm/agent/build.gradle

@@ -63,6 +63,10 @@ thirdPartyAudit {
   )
 }
 
+tasks.named('generateMetadataFileForNebulaPublication') {
+  dependsOn prepareAgent
+}
+
 tasks.named('validateNebulaPom') {
   dependsOn prepareAgent
 }
@@ -75,3 +79,7 @@ tasks.test {
 tasks.check {
   dependsOn test
 }
+
+tasks.named('assemble') {
+  dependsOn prepareAgent
+}

libs/agent-sm/agent/src/main/java/org/opensearch/javaagent/Agent.java

@@ -101,7 +101,7 @@ private static AgentBuilder createAgentBuilder(Instrumentation inst) throws Exce
         final AgentBuilder agentBuilder = new AgentBuilder.Default(byteBuddy).with(AgentBuilder.InitializationStrategy.NoOp.INSTANCE)
             .with(AgentBuilder.RedefinitionStrategy.REDEFINITION)
             .with(AgentBuilder.TypeStrategy.Default.REDEFINE)
-            .ignore(ElementMatchers.none())
+            .ignore(ElementMatchers.nameContains("$MockitoMock$")) /* ingore all Mockito mocks */
             .type(systemType)
             .transform(socketTransformer)
             .type(pathType.or(fileChannelType))

libs/build.gradle

@@ -41,20 +41,21 @@ subprojects {
    */
   project.afterEvaluate {
     if (!project.path.equals(':libs:agent-sm:agent')) {
-      configurations.all { Configuration conf ->
-        dependencies.matching { it instanceof ProjectDependency }.all { ProjectDependency dep ->
-          Project depProject = project.project(dep.path)
-          if (depProject != null
-            && (false == depProject.path.equals(':libs:opensearch-core') &&
-                false == depProject.path.equals(':libs:opensearch-common'))
-            && depProject.path.startsWith(':libs')) {
-            throw new InvalidUserDataException("projects in :libs "
-              + "may not depend on other projects libs except "
-              + ":libs:opensearch-core or :libs:opensearch-common but "
-              + "${project.path} depends on ${depProject.path}")
+        configurations.all { Configuration conf ->
+          dependencies.matching { it instanceof ProjectDependency }.all { ProjectDependency dep ->
+            Project depProject = project.project(dep.path)
+            if (depProject != null
+              && (false == depProject.path.equals(':libs:opensearch-core') &&
+                  false == depProject.path.equals(':libs:opensearch-common')&&
+                  false == depProject.path.equals(':libs:agent-sm:agent-policy'))
+              && depProject.path.startsWith(':libs')) {
+              throw new InvalidUserDataException("projects in :libs "
+                + "may not depend on other projects libs except "
+                + ":libs:opensearch-core, :libs:agent-sm:agent-policy or :libs:opensearch-common but "
+                + "${project.path} depends on ${depProject.path}")
+            }
           }
         }
-      }
     }
   }
 }

libs/secure-sm/build.gradle

@@ -31,6 +31,7 @@ apply plugin: 'opensearch.publish'
 
 dependencies {
   // do not add non-test compile dependencies to secure-sm without a good reason to do so
+  api project(":libs:agent-sm:agent-policy")
 
   testImplementation "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${versions.randomizedrunner}"
   testImplementation "junit:junit:${versions.junit}"