Description
Describe the bug
OpenSearch container does not do a graceful shutdown when a termination signal (SIGTERM/SIGINT) is sent.
This issue was discovered when running the official docker image within Kubernetes, where a rollout would cause the container to be terminated forcefully after terminationGracePeriodSeconds
instead of catching the SIGTERM
signal sent and shutting down cleanly.
This could be fixed by using a proper init process (like tini
) as PID 1 instead of a shell, which would register signal handlers for these termination signals and allow for graceful termination. An example for this change is in the gist below.
Resources with more details on this issue:
- https://hynek.me/articles/docker-signals/
- https://engineeringblog.yelp.com/2016/01/dumb-init-an-init-for-docker.html
To Reproduce
Steps to reproduce the behavior:
- Run an OpenSearch container using the official docker image (tested on both 1.3.7 and 2.5.0).
- Send a
SIGTERM
orSIGINT
to the container and it will not terminate.
Gist here with more detailed reproduction/remediation steps: OpenSearch container termination signal handling
Expected behavior
The container should listen for these termination signals and shutdown gracefully.
Plugins
N/A
Host/Environment (please complete the following information):
- OS: Ubuntu 22.04
- Docker: 20.10.16
- OpenSearch versions: 1.3.7, 2.5.0
Additional context
Add any other context about the problem here.