What happened?
Unsloth Studio keeps the last successful llama.cpp load intent so it can restart a
llama-server process that dies between requests. However, every independent request
gets a fresh generation retry. When each replacement process also dies before completing
a generation, each new request replays the same load intent again with no terminal state.
This is a deterministic supervisor-path reproduction. I did not observe or induce a
production OOM incident, and this report does not claim one.
Environment
- Surface: Unsloth Studio, llama.cpp backend
- Source:
unslothai/unsloth main at 45a371ce0b02fc7eeb47786c285659672a2b53ea
- OS: macOS 26.6.2
- Reproducer: lightweight Python child processes; no GPU or model download required
Steps to reproduce
- Instantiate
LlamaCppBackend with a saved GgufLoadIntent.
- Start a lightweight Python child and terminate it with
SIGKILL (observed return code
-9).
- Replace
load_model with a deterministic harness that records the replay, starts a
replacement child, kills that child with SIGKILL, and reports that startup succeeded.
- Call the existing
_respawn_if_dead() path for five independent requests.
The pre-fix result is five reloads for five requests:
replayed = [true, true, true, true, true]
load_model calls = 5
The existing one-retry-per-generation guard does not bound this sequence because the guard
is recreated for every independent request.
Expected behavior
Automatic recovery should remain available for transient deaths, but consecutive
replacement deaths without a completed generation should eventually reach an explicit
terminal state. A successful, terminal generation should restore the recovery budget for a
future independent failure.
Proposed correction
I have a focused patch that:
- caps automatic replay at three consecutive respawns;
- issues a one-shot recovery token tied to the exact attempt, replacement process, and
unload epoch;
- restores the budget only when that recovery reaches the exact
data: [DONE] terminal
event;
- prevents a stale completion or unrelated request from resetting a newer recovery chain.
Proof collected on the patch:
- regression tests: 5 passed;
- adjacent Studio tests: 76 passed, 0 failed, 0 skipped;
- real-process SIGKILL harness: 1 passed, with three reloads followed by two refusals;
- Ruff on the two changed files: passed;
git diff --check: passed.
No prompts, model data, credentials, or private runtime diagnostics are included.
What happened?
Unsloth Studio keeps the last successful llama.cpp load intent so it can restart a
llama-serverprocess that dies between requests. However, every independent requestgets a fresh generation retry. When each replacement process also dies before completing
a generation, each new request replays the same load intent again with no terminal state.
This is a deterministic supervisor-path reproduction. I did not observe or induce a
production OOM incident, and this report does not claim one.
Environment
unslothai/unslothmain at45a371ce0b02fc7eeb47786c285659672a2b53eaSteps to reproduce
LlamaCppBackendwith a savedGgufLoadIntent.SIGKILL(observed return code-9).load_modelwith a deterministic harness that records the replay, starts areplacement child, kills that child with
SIGKILL, and reports that startup succeeded._respawn_if_dead()path for five independent requests.The pre-fix result is five reloads for five requests:
The existing one-retry-per-generation guard does not bound this sequence because the guard
is recreated for every independent request.
Expected behavior
Automatic recovery should remain available for transient deaths, but consecutive
replacement deaths without a completed generation should eventually reach an explicit
terminal state. A successful, terminal generation should restore the recovery budget for a
future independent failure.
Proposed correction
I have a focused patch that:
unload epoch;
data: [DONE]terminalevent;
Proof collected on the patch:
git diff --check: passed.No prompts, model data, credentials, or private runtime diagnostics are included.