Make kubernetes pods start immediately if it crashes

[mekya]

[muratugureminoglu]

I did this test both on a VM and in a Kubernetes environment.

1. The last time an hs_err_pid*.log file was generated on meet.antmedia.io, AMS became unreachable. I tried to reproduce this issue on my end and successfully generated an hs_err_pid file using the command kill -11 $(pgrep java). However, AMS restarted itself automatically a few seconds after the crash. I believe we need the issue to occur again to investigate why AMS didn’t restart during the last crash.

2. As shown in the configuration here, if port 5080 does not respond, the pod should restart automatically. In the incident described in point 1, port 5080 was still accessible.

In summary, resolving this scenario requires waiting for the issue to reoccur. I was able to generate hs_err_pid files in over 20 crashes, but the exact behavior observed on meet.antmedia.io did not occur. In every case during my tests, AMS restarted automatically.

[muratugureminoglu]

#6895

[muratugureminoglu]

The hs_err file is the HotSpot JVM error dump, and since it is generated by java, Kubernetes expects it to be completely closed, so the restart time of the pod varies accordingly. I searched for a limit on this file on the java side, but I couldn't find a solution. I can configure Kubernetes to only restart the pod as soon as this file is created.