Skip to content

[BUG] OpenSearch container does not allow for graceful termination #3229

Closed
@shikharbhardwaj

Description

@shikharbhardwaj

Describe the bug

OpenSearch container does not do a graceful shutdown when a termination signal (SIGTERM/SIGINT) is sent.
This issue was discovered when running the official docker image within Kubernetes, where a rollout would cause the container to be terminated forcefully after terminationGracePeriodSeconds instead of catching the SIGTERM signal sent and shutting down cleanly.

This could be fixed by using a proper init process (like tini) as PID 1 instead of a shell, which would register signal handlers for these termination signals and allow for graceful termination. An example for this change is in the gist below.

Resources with more details on this issue:

To Reproduce
Steps to reproduce the behavior:

  1. Run an OpenSearch container using the official docker image (tested on both 1.3.7 and 2.5.0).
  2. Send a SIGTERM or SIGINT to the container and it will not terminate.

Gist here with more detailed reproduction/remediation steps: OpenSearch container termination signal handling

Expected behavior
The container should listen for these termination signals and shutdown gracefully.

Plugins
N/A

Host/Environment (please complete the following information):

  • OS: Ubuntu 22.04
  • Docker: 20.10.16
  • OpenSearch versions: 1.3.7, 2.5.0

Additional context
Add any other context about the problem here.

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions