OkHttp Connection Leaks

[lordpengwin]

I've been trying to create an operator using the quarkus-operator-sdk and see a lot of Warning about OkHttp ConnectionPool leaks. I also see problems where my operator simply stops working after a certain period of time with errors about resource age. To rule out my I tried testing the samples in this repo and see similar problems:

---

--/ __ / / / / _ | / _ / /// / / / __/
-/ // / // / __ |/ , / ,< / // /\ \
--___// |//|//||_//
2021-11-04 13:12:29,620 INFO [io.jav.ope.Operator] (main) Registered Controller: 'jokecontroller' for CRD: 'class io.quarkiverse.operatorsdk.samples.joke.JokeRequest' for namespace(s): [default]
2021-11-04 13:12:29,636 INFO [io.qua.ope.run.AppEventListener] (main) Quarkus Java Operator SDK extension 2.0.1-SNAPSHOT (commit: 7acc366 on branch: main) built on Thu Nov 04 12:40:31 GMT 2021
2021-11-04 13:12:29,637 INFO [io.jav.ope.Operator] (main) Operator SDK 1.9.11 (commit: 0287c0c) built on Fri Oct 29 >16:12:51 GMT 2021 starting...
2021-11-04 13:12:29,637 INFO [io.jav.ope.Operator] (main) Client version: 5.8.0
2021-11-04 13:12:30,018 INFO [io.jav.ope.Operator] (main) Server version: 1.20+
2021-11-04 13:12:30,421 INFO [io.quarkus] (main) quarkus-operator-sdk-samples 2.0.1-SNAPSHOT on JVM (powered by Quarkus 2.4.0.Final) started in 3.065s. Listening on: http://0.0.0.0:8080
2021-11-04 13:12:30,423 INFO [io.quarkus] (main) Profile prod activated.
2021-11-04 13:12:30,423 INFO [io.quarkus] (main) Installed features: [CSVGeneration, cdi, kubernetes, kubernetes-client, openshift-client, operator-sdk, rest-client, rest-client-jackson, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, vertx]
2021-11-04 13:13:16,688 WARN [io.fab.kub.cli.int.VersionUsageUtils] (EventHandler-jokecontroller) The client is using resource type 'jokerequests' with unstable version 'v1alpha1'
2021-11-04 13:13:17,098 WARN [io.fab.kub.cli.int.VersionUsageUtils] (EventHandler-jokecontroller) The client is using resource type 'jokes' with unstable version 'v1alpha1'
2021-11-04 13:17:29,917 WARNING [okh.OkHttpClient] (OkHttp ConnectionPool) A connection to https://10.100.0.1/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE);

Is there anything that can be set to manage this connection?

[metacosm]

There is a known issue with the fabric8 client which should be addressed in an upcoming version.

[lordpengwin]

Great! I assume you are referencing this fabric8io/kubernetes-client#3561
Thanks!

[gsmet]

@metacosm FWIW, I fixed a similar issue in GitHub API. The javadoc says closing the InputStream should close the ResponseBody but it didn't and I did something like that to fix it:
https://github.com/hub4j/github-api/pull/984/files

Not sure if it might help but maybe it's just a quick fix for you if you apply a similar pattern.

(The issue might be entirely different, I haven't checked)

[metacosm]

@gsmet thank you Guillaume but the problem in this case is that this occurs within the fabric8 client so it's not something we can control from this extension, unfortunately, as we don't access the OkHttp code directly.

[sclorng]

With fabric 5.11.1, quarkus 2.6.1, and operator sdk 2.0.2, I have an error each time I used an injected KubernetesClient cannot make a new request because the previous response is still open: please call response.close() but I do not make any previous call on my own.

Can it be linked to this issue ? Seems it is not very thread safe.

2022-01-03 14:29:14,082 ERROR [io.jav.ope.pro.EventDispatcher] (EventHandler-jwker) Error during event processing ExecutionScope{events=[{ class=io.javaoperatorsdk.operator.processing.event.internal.TimerEvent, customResourcesSelector=UIDMatchingPredicate{uid='779d51ab-977c-4b2c-a8d8-ae34773a120d'}, eventSource=io.javaoperatorsdk.operator.processing.event.internal.TimerEventSource@72d0cc08 }], customResource uid: 779d51ab-977c-4b2c-a8d8-ae34773a120d, version: 375112174} failed.: java.lang.IllegalStateException: cannot make a new request because the previous response is still open: please call response.close()

I'm just updating a secret with kubernetesClient.secrets().inNamespace(resource.getSpec().getSecret().getNamespace()).createOrReplace(secret);

[metacosm]

It's a known issue: fabric8io/kubernetes-client#3697

[scrocquesel]

operator-framework/java-operator-sdk#781

[scrocquesel]

5.11.2 fixes my issue and it should do so for this one. (Dependabot should make the PR soon)

What about a quarkus-operator-sdk 2.0.3 with quarkus 2.6.1 and kubernetes client 5.11.2 ?

[metacosm]

Yep. :)

[sclorng]

Yep. :)

@metacosm may I ask for a quarkus operator sdk version now that the JOSK wa bumped.

[metacosm]

I was waiting for Quarkus 2.6.2 but I guess I could release a new version now.

[scrocquesel]

I was waiting for Quarkus 2.6.2 but I guess I could release a new version now.

Oh yes, it has the fix also for smallrye NPE. Definitely worth waiting for me