[Bug]: Shutdown hook in RyukResourceReaper prevents a graceful shutdown with spring framework

[jobayle]

Module

Core

Testcontainers version

1.19.7

Using the latest Testcontainers version?

Yes

Host OS

Linux, Windows

Host Arch

x86_64

Docker version

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:18:56 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.29.0 (145265)
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:01 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

Dears,

With the introduction of this feature: #7717

The ryuk instance is terminated by a shutdown hook, therefore all containers are closed very early.

This prevents other shutdown hooks (eg: those set by the spring framework) to perform actions needed for a graceful shutdown that still require containers to be up.
Due to a retry mechanism in Spring's shutdown hook, it greatly increases the overall time to run tests!

Could you please revert this change or introduce a mean to disable this shutdown hook, thanks!

Relevant log output

No response

Additional Information

No response

[s-jepsen]

Will this be fixed any time soon?

[EAlf91]

Also relevant for #7871 there was a solution proposed in the issue as well

[jobayle]

Hi @EAlf91, I cannot find the solution in the linked issue, could you please add a link to the specific comment? thanks!

[EAlf91]

@jobayle It's mentioned here: #7871 (comment) but it would require a change I guess. The author made the hook configurable via spring properties: alex-arana@a0fcbdf

I didn't try it myself and didn't see any way to configure this without using this fork. I'd highly appreciate to see a revert of #7717 or maybe using this way to configure it via spring properties

[jobayle]

Thanks @EAlf91, but imho, this is not a viable solution if we have to maintain a patched version of testcontainers.

[s-jepsen]

Any progress on this issue?

[jpalaz]

@eddumelendez @bsideup @kiview Mention you all because you were working on PR which lead to this issue: #7717

Could you please take a brief look on this 🙏

[kiview]

Hey @jobayle,

Due to a retry mechanism in Spring's shutdown hook, it greatly increases the overall time to run tests!

This is of course a situation we would like to solve. What absolute amount of time are we talking abut here?

I am not sure #8732 is the right approach though, since this would still require all affected users, to apply this config option (not ideal). We could make configuration more programmatically accessible, but this would also require downstream integration changes by the Spring project.

Since a considerable amount of Testcontainers users are also Spring users, I think reverting the change and accept the containers lingering for some time after process termination can be the better DX, WDYT @eddumelendez?

[jobayle]

Hi @kiview, sure:
Here's the first and last log of the SpringApplicationShutdownHook:

16:15:25.372 [SpringApplicationShutdownHook] INFO - UPDATING MACHINE CONFIGURATION STATUS=SHUTDOWN
[...] lots of connection errors with DB/redis/...
16:16:27.222 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-13 - Shutdown completed.

So roughly a whole minute!

By enabling container reuse to simulate how things worked before #7717, the shutdown is almost instant:

16:38:34.259 [SpringApplicationShutdownHook] INFO  - UPDATING MACHINE CONFIGURATION STATUS=SHUTDOWN
16:38:34.749 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-3 - Shutdown initiated...
16:38:34.751 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-3 - Shutdown completed.
16:38:34.753 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Shutdown initiated...
16:38:34.755 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Shutdown completed.
16:38:34.755 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown initiated...
16:38:34.758 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown completed.
16:38:34.806 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-9 - Shutdown initiated...
16:38:34.807 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-9 - Shutdown completed.
16:38:34.810 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-8 - Shutdown initiated...
16:38:34.813 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-8 - Shutdown completed.
16:38:34.813 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-7 - Shutdown initiated...
16:38:34.816 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-7 - Shutdown completed.
16:38:34.856 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-15 - Shutdown initiated...
16:38:34.857 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-15 - Shutdown completed.
16:38:34.860 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-14 - Shutdown initiated...
16:38:34.862 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-14 - Shutdown completed.
16:38:34.862 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-13 - Shutdown initiated...
16:38:34.864 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-13 - Shutdown completed.

Yes, I'd really love a fix or a revert, thanks 🙂

[kiview]

Thanks for sharing @jobayle, 1 minute is really not acceptable, will sync with @eddumelendez on how to best proceed with this and resolve the situation 👍.

[xak2000]

The registering of shutdown hook also has other problems, such as classloader issues in some specific use-cases. Considering this, I vote for reverting the change that introduced the shutdown hook. Or at least made it configurable (and 'off' by default).

The solution of this problem will also solve the problem of using Testcontainers from Maven (e.g. #7923, #1454, testcontainers/testcontainers-jooq-codegen-maven-plugin#37) if shutdown hook will not be registered by default or at least configurable.