Internal hosted UC not supported

[sagar-khanna]

We are hosting update center in Nexus which has a self-signed certificate so when I run this command it fails with the following error:

io.jenkins.tools.pluginmanager.impl.UpdateCenterInfoRetrievalException: Error getting update center json at io.jenkins.tools.pluginmanager.impl.PluginManager.getJson(PluginManager.java:578) at io.jenkins.tools.pluginmanager.impl.PluginManager.getUCJson(PluginManager.java:597) at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:146) at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:113) at io.jenkins.tools.pluginmanager.cli.Main.main(Main.java:63) Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268) at java.net.URL.openStream(URL.java:1068) at org.apache.commons.io.IOUtils.toString(IOUtils.java:2795) at io.jenkins.tools.pluginmanager.impl.PluginManager.getJson(PluginManager.java:572) ... 4 more Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:450) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:317) at sun.security.validator.Validator.validate(Validator.java:262) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) ... 19 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:445) ... 25 more Error getting update center json

I tried adding our ca certificate to /usr/local/share/certificates and then running update-ca-certificates but it still doesn't work.
Perhaps we need a flag to ignore self-signed certificates eg when using the script I could pass CURL_OPTS -k

[timja]

Hi

You need to determine which truststore is being used by your Java installation and ensure it's updated with your internal cert, (or get a trusted cert via something like let's encrypt normally makes things a lot easier).

Here's a post about how to locate it: https://www.ibm.com/support/pages/how-identify-ssltls-keystore-and-truststore-being-used-java%E2%84%A2-application

[sagar-khanna]

Thanks @timja for a quick response.

I have it in my Dockerfile like this:

RUN jenkins-plugin-cli --verbose --jenkins-update-center https://<INTERNAL_FQDN>/repository/jenkins-plugins/update-center.json -f /usr/share/jenkins/plugins.txt

So do I need to use keytool to import my root ca into the Java's trust store inside the Jenkins docker image?

[timja]

Yes you do, or add your custom truststore to your image

[sagar-khanna]

I'll try keystore and get back shortly.

I have already tried adding custom trust store like below but it didn't work:
COPY <INTERNAL_ROOT_CA>.pem /usr/local/share/ca-certificates/ RUN chmod 644 /usr/local/share/ca-certificates/<INTERNAL_ROOT_CA>.pem && update-ca-certificates

Additionally is it possible to add a feature that sets a flag to the cli to ignore self-signed certificates?

[timja]

I would rather document the proper solution, so once you get it working would be good if you can report back

[sagar-khanna]

Still fails

Step 10/21 : RUN keytool -import -trustcacerts -alias "<REDACTED>" -file /usr/local/share/ca-certificates/<REDACTED>.pem -keystore /usr/local/openjdk-8/jre/lib/security/cacerts -storepass changeit -noprompt
 ---> Running in bd55ee69a176
Certificate was added to keystore
Removing intermediate container bd55ee69a176
 ---> 3f3d42dcd4ed
Step 11/21 : RUN keytool -list -keystore /usr/local/openjdk-8/jre/lib/security/cacerts -storepass changeit | grep -i <REDACTED>
 ---> Running in 6cce19a2fe76
<REDACTED>, Nov 16, 2020, trustedCertEntry,
Removing intermediate container 6cce19a2fe76

I'm out of ideas now :-(

Wouldnt it be a proper solution to add a flag to the jenkins-plugin-cli tool to ignore SSL warning for self-signed cert?

[timja]

have you verified that's the trust store actually in use?

you can also for using your own truststore as well.

[sagar-khanna]

Ignore the above error. The CA certificate was invalid, used the correct one and now its past that problem.

However now I'm stuck here:
Cache miss for: plugin-versions

Basically I'm not hosting this https://updates.jenkins.io/plugin-versions.json, I am using https://github.com/jenkinsci/juseppe to generate the release-history and update-centre json files and it doesn't support plugin-versions.json file at the moment.

Is there a way to skip experimental, incrementals and plugin info ?

[timja]

It looks like it should work if you override the configuration and point it at your update-center.json

is it not release-history.json btw? just looking at the docs?

[sagar-khanna]

No, in release-history.json the key is called release-history whereas for plugins.version its called plugins.
The content is also different, release-history gives info about the releases itself whereas plugins gives history of the plugin versions.

I just did an override pointing to the update-centre.json for all three URLs and now it fails because it can't find certain keys in the data:
org.json.JSONException: JSONObject["requiredCore"] not found.

[timja]

💩 any thoughts @oleg-nenashev / @daniel-beck, is it better to add to juseppe these files or should we handle that these files may not be there?

[daniel-beck]

Depends; the cleaner solution would be an improvement to Juseppe IMO, while not relying on this file would be more compatible.

FWIW release-history is pretty terrible: It doesn't list more than one release of a given plugin per day, i.e. some releases are just not listed. So seems unsuitable beyond use as a data source for an automated Twitter account.

[sagar-khanna]

I copied the plugin-versions.json from update.jenkins.io and put it in our Nexus repo however still getting the exception (given below) for some plugins.
The exception doesn't provide the name of the plugin so I put the plugins list in a for loop and echo the plugin name before the install command however because it runs in async mode the output is all jumbled up so I can't really tell from 150+ plugins that I'm trying to install for which plugin requiredCore is missing in plugins-version.json if at all.

org.json.JSONException: JSONObject["requiredCore"] not found.
        at org.json.JSONObject.get(JSONObject.java:572)
        at org.json.JSONObject.getString(JSONObject.java:859)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.getPluginDependencyJsonArray(PluginManager.java:629)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveDependenciesFromJson(PluginManager.java:742)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveDirectDependencies(PluginManager.java:794)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveRecursiveDependencies(PluginManager.java:831)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.findPluginsAndDependencies(PluginManager.java:490)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:153)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:113)
        at io.jenkins.tools.pluginmanager.cli.Main.main(Main.java:63)
JSONObject["requiredCore"] not found.```

[timja]

I think we need to add this to Juseppe really

[sagar-khanna]

I think we need to add this to Juseppe really
Does that mean orgs not using juseppe but using this tool should work out a way to make sure these files are available?

Also would you like for me to raise an issue in the juseppe repo given that we should be generating these files there anyways?

We are kind of blocked from rolling out Jenkins using Docker at the moment because of this, I'll try to use a workaround but not 100% sure at the moment if it works.