How can we handle pre shutdown?

[Takashi-kun]

Hello,

Currently, when uvicorn's code receives a signal, the shutdown process begins immediately. This process executes server.close right after the signal is received, which prevents implementing a graceful shutdown flow, such as failing a health check from a load balancer before shutting down.

• https://github.com/encode/uvicorn/blob/master/uvicorn/server.py#L263-L269

To properly handle requests from a load balancer, I'd like to be able to trigger an event like preShutdown after receiving SIGTERM/SIGINT to modify the behavior of the health check handler before closing server.

For example, consider the health check implementation here:

app = FastAPI()
is_shutting_down = False


def pre_shutdown_handler():
    global is_shutting_down
    is_shutting_down = True

@app.get("/health")
async def health():
    global is_shutting_down
    if is_shutting_down:
        return Response(
            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
            content="service unavailable",
        )
    return Response(status_code=status.HTTP_200_OK, content="ok")

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(
        "src.main:app",
        host="0.0.0.0",
        port=PORT,
        log_level="info",
        reload=True,
        timeout_graceful_shutdown=10,
    )

Is it possible to trigger an event such as preShutdown? Or is there such a feature on the roadmap?

Thank you for your time.

[nggit]

There is no way other than patching. There is nothing in the ASGI spec that regulates that.

[Takashi-kun]

@nggit Thanks for your sample code.
In my understanding is correct, since the startup event is called after prepared built-in uvicorn's signal handler, the my original signal_hook is executed after built-in handling like this.

• https://github.com/encode/uvicorn/blob/master/uvicorn/server.py#L68-L87

Skipping virtualenv creation, as specified in config file.
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
^CINFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [1]
Got SIGINT. Do your thing here...
Got SIGINT. Do your thing here...

[nggit]

Correct. Because in practice the server already sets the handler before the ASGI event startup. It should work universally on any ASGI server. But my print() above is actually not a good example to demonstrate the correct order.

If you are curious why you received two SIGINTs, in my experience:

1. It's automatic propagation from the main process (Python / system level behavior).
2. It's coded somewhere to send a SIGINT to the child.

The reason why no. 2 is done is generally for predictability. Number 1 (relying on automatic) is not always propagated.

So you have to make sure your handler tolerates more than one call.

[Takashi-kun]

@nggit Thanks for your reply and sorry for late.

I found interesting things.
I add time.sleep(10) before executing orignal_handlers to configure the period to separate the instance from load balancer like this.

@app.on_event("startup")
async def startup():
    loop = asyncio.get_running_loop()
    original_handlers = {
        signal.SIGINT: signal.getsignal(signal.SIGINT),
        signal.SIGTERM: signal.getsignal(signal.SIGTERM),
    }

    def signal_hook(signum):
        global is_shutting_down
        is_shutting_down = True
        print(f"{signum} received. Shutting down in 10 seconds...")
        time.sleep(10)
        if callable(original_handlers[signum]):
            original_handlers[signum](signum, None)

    loop.add_signal_handler(signal.SIGINT, signal_hook, signal.SIGINT)
    loop.add_signal_handler(signal.SIGTERM, signal_hook, signal.SIGTERM)

@app.get("/health")
async def health():
    global is_shutting_down
    if is_shutting_down:
        return Response(
            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
            content="service unavailable",
        )
    return Response(status_code=status.HTTP_200_OK, content="ok")

the application waits 10s before shutting down, it's expected.

root@bcc2235b1f74:/app# poetry run uvicorn src.main:app --host 0.0.0.0 --port 8080
INFO:     Started server process [14]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     192.168.65.1:35116 - "GET /health HTTP/1.1" 200 OK
^C2 received. Shutting down in 10 seconds...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [14]
2 received. Shutting down in 10 seconds...

but the curl hangs 10s as well, it's not expected.
I expected to return 503 soon and add grace time before shutting down when signal received

$ time curl localhost:8080/health
okcurl localhost:8080/health  0.01s user 0.01s system 43% cpu 0.039 total
$ time curl localhost:8080/health
service unavailablecurl localhost:8080/health  0.01s user 0.01s system 0% cpu 9.241 total

Do you have any idea for this?

[nggit]

Instead of blocking sleep, you can delay the original handler call with loop.call_later:

        if callable(original_handlers[signum]):
            loop.call_later(10, original_handlers[signum], signum, None)

[Takashi-kun]

@nggit Thank you~!!

It works well as expectedly:

1. returns 503 when signal received
2. 10s later, the server shutdown process started
   • during received signal and starting shutdown, server listens
3. change healthcheck HTTP status code after signal

Let me close this discussion, thanks so much!!!