Remove JCenter Dependencies

[peternied]

See the OpenSearch Forum Announcement

Execution failed for task ':buildSrc:compileJava'.
> Could not resolve all files for configuration ':buildSrc:compileClasspath'.
   > Could not resolve org.apache.maven:maven-model:3.6.2.
     Required by:
         project :buildSrc

Reproduce with
./build.sh manifests/1.2.4/opensearch-1.2.4.yml --component OpenSearch --snapshot

Full Error stack

> Task :buildSrc:reaper:check UP-TO-DATE
> Task :buildSrc:reaper:build
> Task :buildSrc:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':buildSrc:compileJava'.
> Could not resolve all files for configuration ':buildSrc:compileClasspath'.
   > Could not resolve org.apache.maven:maven-model:3.6.2.
     Required by:
         project :buildSrc
      > Could not resolve org.apache.maven:maven-model:3.6.2.
         > Could not parse POM https://jcenter.bintray.com/org/apache/maven/maven-model/3.6.2/maven-model-3.6.2.pom
            > Could not resolve org.apache.maven:maven:3.6.2.
               > Could not resolve org.apache.maven:maven:3.6.2.
                  > Could not parse POM https://jcenter.bintray.com/org/apache/maven/maven/3.6.2/maven-3.6.2.pom
                     > Could not resolve org.apache.maven:maven-parent:33.
                        > Skipped due to earlier error
   > Could not resolve org.apache.ant:ant-launcher:1.10.9.
     Required by:
         project :buildSrc > org.apache.ant:ant:1.10.9
      > Skipped due to earlier error
   > Could not resolve com.perforce:p4java:2015.2.1365273.
     Required by:
         project :buildSrc > com.netflix.nebula:gradle-info-plugin:7.1.3
      > Skipped due to earlier error
   > Could not resolve com.netflix.nebula:gradle-contacts-plugin:5.1.0-rc.1.
     Required by:
         project :buildSrc > com.netflix.nebula:gradle-info-plugin:7.1.3
      > Skipped due to earlier error
   > Could not resolve org.eclipse.jgit:org.eclipse.jgit:5.0.1.201806211838-r.
     Required by:
         project :buildSrc > com.netflix.nebula:gradle-info-plugin:7.1.3
      > Skipped due to earlier error
   > Could not resolve org.tmatesoft.svnkit:svnkit:1.8.12.
     Required by:
         project :buildSrc > com.netflix.nebula:gradle-info-plugin:7.1.3
      > Skipped due to earlier error
   > Could not resolve org.apache.rat:apache-rat-core:0.13.
     Required by:
         project :buildSrc > org.apache.rat:apache-rat:0.13
      > Skipped due to earlier error
   > Could not resolve org.apache.rat:apache-rat-tasks:0.13.
     Required by:
         project :buildSrc > org.apache.rat:apache-rat:0.13
      > Skipped due to earlier error
   > Could not resolve org.slf4j:slf4j-api:1.7.25.
     Required by:
         project :buildSrc > com.networknt:json-schema-validator:1.0.36
      > Skipped due to earlier error
   > Could not resolve org.apache.commons:commons-lang3:3.5.
     Required by:
         project :buildSrc > com.networknt:json-schema-validator:1.0.36
      > Skipped due to earlier error
   > Could not resolve org.jruby.joni:joni:2.1.31.
     Required by:
         project :buildSrc > com.networknt:json-schema-validator:1.0.36
      > Skipped due to earlier error
   > Could not resolve com.fasterxml.jackson.core:jackson-annotations:2.12.5.
     Required by:
         project :buildSrc > com.fasterxml.jackson.core:jackson-databind:2.12.5
         project :buildSrc > com.fasterxml.jackson.core:jackson-databind:2.12.5 > com.fasterxml.jackson:jackson-bom:2.12.5
      > Skipped due to earlier error
   > Could not resolve com.fasterxml.jackson.core:jackson-core:2.12.5.
     Required by:
         project :buildSrc > com.fasterxml.jackson.core:jackson-databind:2.12.5
         project :buildSrc > com.fasterxml.jackson.core:jackson-databind:2.12.5 > com.fasterxml.jackson:jackson-bom:2.12.5
      > Skipped due to earlier error

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13s
2022-01-12 17:24:33 ERROR    Error building OpenSearch, retry with: ./build.sh manifests/1.2.4/opensearch-1.2.4.yml --component OpenSearch --snapshot
Traceback (most recent call last):
  File "./src/run_build.py", line 78, in <module>
    sys.exit(main())
  File "./src/run_build.py", line 66, in main
    builder.build(build_recorder)
  File "/var/jenkins/workspace/distribution-build-opensearch/src/build_workflow/builder_from_source.py", line 50, in build
    self.git_repo.execute(build_command)
  File "/var/jenkins/workspace/distribution-build-opensearch/src/git/git_repository.py", line 83, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bash /var/jenkins/workspace/distribution-build-opensearch/scripts/components/OpenSearch/build.sh -v 1.2.4 -p linux -a x64 -s true -o builds' returned non-zero exit status 1.

What’s the problem?

JCenter, which is a repository used in our builds, ceased operations today. Gradle was pulling dependencies from JCenter, and some repos have a direct dependency on it. This broke that the build process for OpenSearch engine, all branches so that the last successful build of 1.3 was at 7:03 AM PT Jan 12th.

To Fix:

1. Remove any direct dependency on jcenter(). This would appear in a repositories block in build.gradle. Example.
2. If your plugin depends on a gradle plugin that cannot currently be downloaded from https://plugins.gradle.org/ because its dependencies are failing to pull from jcenter. A workaround is to define a repositories block at the top of your settings.gradle file that allows us to control the order and first search in maven central.

pluginManagement {
  repositories {
    mavenCentral()
    gradlePluginPortal()
  }
}

Remove JCenter Tasks

• Remove jcenter repository notifications#373 - Notification team is not taking changes at this time.
• There are zero references to jcenter repositories in code, query.
• Remove jcenter repository alerting#278
• Remove jcenter repository asynchronous-search#89
• Remove jcenter repository data-prepper#1062
• Remove jcenter repository common-utils#110
• Remove jcenter repository reporting#276
• Remove jcenter repository perftop#34
• Remove jcenter repository dashboards-notebooks#110
• Remove jcenter repository data-prepper#870
• Remove jcenter repository anomaly-detection#357
• Remove jcenter repository k-NN#260
• Remove jcenter repository sql#377
• Remove jcenter repository geospatial#16
• Remove jcenter repository cross-cluster-replication#289
• Remove jcenter repository ml-commons#120
• Remove jcenter repository observability#372
• Remove jcenter repository security#1625
• Remove jcenter repository reference in blog post project-website#698
• [Backport 1.x] Replace JCenter with Maven Central. (#1057) OpenSearch#1892

[dblock]

jcenter bintray is down, https://isdown.app/integrations/bintray

Do note that jcenter bintray has been sunset, even though it will remain online, we need a campaign to remove our dependency on it.

[mch2]

Been looking at this a bit -

From a
./gradlew build --debug

2022-01-12T10:28:31.122-0800 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.RepositoryChainComponentMetaDataResolver] Attempting to resolve component for org.yaml:snakeyaml:1.26 using repositories [BintrayJCenter, opensearch-downloads, opensearch-downloads2, opensearch-snapshots, elasticsearch-downloads, elasticsearch-snapshots, MavenRepo, MavenLocal, MavenRepo2, maven]

Most of these are coming from DistributionDownloader, but I don't see JCenter defined there. I think this is being brought in by gradle.

From https://blog.gradle.org/jcenter-shutdown -

The Gradle Plugin Portal implicitly mirrors JCenter currently. If you’re using the Plugin Portal (via gradlePluginPortal() or the URL plugins.gradle.org/m2) to resolve your application’s dependencies, you may be relying on JCenter. You should avoid using the Plugin Portal as a repository, except for Gradle plugin projects.

We are using this here: https://github.com/opensearch-project/OpenSearch/blob/main/buildSrc/build.gradle#L98

[mch2]

Just noticed that 1.x branch is still explicitly referencing jcenter() here

[dblock]

Thanks for looking into it @mch2, let's fix OpenSearch on all branches, then I'll cascade this issue into plugins as needed to get rid of anything jcenter.

[mch2]

So I initiated that backport to 1.x to remove the explicit jcenter() dep in 1.x, but its still blowing up. I think the isue is in gradlePluginPortal().

Unfortunately without gradlePluginPortal() defined we are unable to resolve a few dependencies:

Could not determine the dependencies of task ':buildSrc:compileGroovy'.

Could not resolve all task dependencies for configuration ':buildSrc:compileClasspath'.
Could not find com.github.jengelman.gradle.plugins:shadow:6.0.0.
Searched in the following locations:
- https://repo.maven.apache.org/maven2/com/github/jengelman/gradle/plugins/shadow/6.0.0/shadow-6.0.0.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project :buildSrc
Could not find com.avast.gradle:gradle-docker-compose-plugin:0.12.1.
Searched in the following locations:
- https://repo.maven.apache.org/maven2/com/avast/gradle/gradle-docker-compose-plugin/0.12.1/gradle-docker-compose-plugin-0.12.1.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project :buildSrc

[owaiskazi19]

Apparently spotless also uses jcenter and it's on main.
Could not GET 'https://jcenter.bintray.com/com/diffplug/spotless/spotless-lib/2.7.0/spotless-lib-2.7.0.module'. Received status code 502 from server: Bad Gateway.
https://github.com/opensearch-project/OpenSearch/runs/4793525694?check_suite_focus=true

[mch2]

Tried an explicit dependency on plugins.gradle.org/m2/ instead of gradlePluginPortal() - looks like the version of spotless we are using is not there so it mirrors to jCenter.

A problem occurred configuring root project 'OpenSearch'.
Could not determine the dependencies of null.
Could not resolve all task dependencies for configuration ':classpath'.
> Could not resolve com.diffplug.spotless:spotless-lib:2.7.0.
Required by:
project : > com.diffplug.spotless:com.diffplug.spotless.gradle.plugin:5.6.1 > com.diffplug.spotless:spotless-plugin-gradle:5.6.1
> Could not resolve com.diffplug.spotless:spotless-lib:2.7.0.
> Could not get resource 'https://plugins.gradle.org/m2/com/diffplug/spotless/spotless-lib/2.7.0/spotless-lib-2.7.0.module'.
> Could not GET 'https://jcenter.bintray.com/com/diffplug/spotless/spotless-lib/2.7.0/spotless-lib-2.7.0.module'. Received status code 502 from server: Bad Gateway

[dlvenable]

@mch2 , Did you try to disable the Gradle module files as noted here? opensearch-project/OpenSearch#1892 (comment) I see that it is trying to load a .module file.

[mch2]

@mch2 , Did you try to disable the Gradle module files as noted here? opensearch-project/OpenSearch#1892 (comment) I see that it is trying to load a .module file.

Thanks @dlvenable for the suggestion. I've tried it but still getting the same trace. With debug enabled I can see plugins.gradle.com redirecting to bintray. Spotless 5.6.1 declares other spotless dependencies that are sourced from bintray, so it redirects there - ex.

    <dependency>
      <groupId>com.diffplug.spotless</groupId>
      <artifactId>spotless-lib</artifactId>
      <version>2.7.0</version>
      <scope>compile</scope>
      <type>jar</type>
      <optional>false</optional>
    </dependency>

2022-01-12T12:04:55.177-0800 [DEBUG] [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection released: [id: 1][route: {s}->https://plugins.gradle.org:443][total available: 2; route allocated: 2 of 20; total allocated: 2 of 20] 2022-01-12T12:04:55.177-0800 [DEBUG] [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection released: [id: 0][route: {s}->https://plugins.gradle.org:443][total available: 2; route allocated: 2 of 20; total allocated: 2 of 20] 2022-01-12T12:04:55.180-0800 [DEBUG] [org.gradle.internal.resource.transport.http.AlwaysFollowAndPreserveMethodRedirectStrategy] Redirect requested to location 'https://jcenter.bintray.com/com/diffplug/spotless/spotless-lib-extra/2.7.0/spotless-lib-extra-2.7.0.pom' 2022-01-12T12:04:55.180-0800 [DEBUG] [org.gradle.internal.resource.transport.http.AlwaysFollowAndPreserveMethodRedirectStrategy] Redirect requested to location 'https://jcenter.bintray.com/com/diffplug/spotless/spotless-lib/2.7.0/spotless-lib-2.7.0.pom'

[mch2]

Updates from gradle - https://status.gradle.com/?_ga=2.113826690.463275434.1641929099-1662752518.1641929099