Surfaced while diagnosing a CI failure in Chat UI, API, Update and GGUF inference. The test defect that made it fail the job is fixed separately in #9711; this issue is the underlying backend behaviour, which is real, is on main, and is NOT fixed.
What happens
The Studio backend stops serving every route for a while, then recovers on its own. Observed three times in one job:
- 10.03s
- 33.2s
- 27.75s (on a different backend instance)
During a window, /api/liveness and /api/health both time out against a 10s client budget. Each window ends on an /api/inference/status whose process_time_ms equals the length of the window, so the request was accepted and then sat there. The process never dies, and the end-of-run liveness check passes.
How often
Over 11 days of that workflow the affected step ran 1711 times and failed once, and on main it is 0 failures in 287 runs. Passing runs report 24 samples, 0 non-200 with a median worst latency of 47.7ms, roughly 200x under the budget. So this is rare, but a 33s total stall is far outside the normal spread rather than the tail of it.
Why it matters even though CI now tolerates it
The launcher watchdog (studio/src-tauri/src/commands.rs) polls every 15s and gives up after 3 consecutive failures, widened to 12 when a probe times out against a generating backend, plus a 30s last-chance probe. A 33s stall is two strikes, so the launcher keeps the backend. That is the right behaviour and is why #9711 relaxes the test to the same rule. But a user on a 33s stall sees an application that is completely unresponsive to every request, with no indication anything is wrong.
Two hypotheses, both refuted
- A blocking call on the event loop. Refuted: the status probe already runs off the loop through
_STATUS_PROBE_EXECUTOR.
- The MLX self-heal thread holding the GIL. Refuted by the logs: the only instance in that job which ran the self-heal had ZERO stalls (max 3363ms over 376 requests), and neither stalled instance ran it at all.
Suggested next step
The mechanism is still unknown and I have not been able to reproduce it without a macOS host, so I have deliberately not guessed at a patch. The most direct way to name it is a py-spy dump taken against a backend while it is stalled, which identifies the GIL holder rather than inferring it. A watchdog thread that dumps on N consecutive slow self-probes would catch it without needing anyone present.
Surfaced while diagnosing a CI failure in
Chat UI, API, Update and GGUF inference. The test defect that made it fail the job is fixed separately in #9711; this issue is the underlying backend behaviour, which is real, is on main, and is NOT fixed.What happens
The Studio backend stops serving every route for a while, then recovers on its own. Observed three times in one job:
During a window,
/api/livenessand/api/healthboth time out against a 10s client budget. Each window ends on an/api/inference/statuswhoseprocess_time_msequals the length of the window, so the request was accepted and then sat there. The process never dies, and the end-of-run liveness check passes.How often
Over 11 days of that workflow the affected step ran 1711 times and failed once, and on main it is 0 failures in 287 runs. Passing runs report
24 samples, 0 non-200with a median worst latency of 47.7ms, roughly 200x under the budget. So this is rare, but a 33s total stall is far outside the normal spread rather than the tail of it.Why it matters even though CI now tolerates it
The launcher watchdog (
studio/src-tauri/src/commands.rs) polls every 15s and gives up after 3 consecutive failures, widened to 12 when a probe times out against a generating backend, plus a 30s last-chance probe. A 33s stall is two strikes, so the launcher keeps the backend. That is the right behaviour and is why #9711 relaxes the test to the same rule. But a user on a 33s stall sees an application that is completely unresponsive to every request, with no indication anything is wrong.Two hypotheses, both refuted
_STATUS_PROBE_EXECUTOR.Suggested next step
The mechanism is still unknown and I have not been able to reproduce it without a macOS host, so I have deliberately not guessed at a patch. The most direct way to name it is a
py-spy dumptaken against a backend while it is stalled, which identifies the GIL holder rather than inferring it. A watchdog thread that dumps on N consecutive slow self-probes would catch it without needing anyone present.