Skip to content

[Studio Bug] Repeated SIGKILL recovery replays the same load intent without a bound #9688

Description

@JulienJBO

What happened?

Unsloth Studio keeps the last successful llama.cpp load intent so it can restart a
llama-server process that dies between requests. However, every independent request
gets a fresh generation retry. When each replacement process also dies before completing
a generation, each new request replays the same load intent again with no terminal state.

This is a deterministic supervisor-path reproduction. I did not observe or induce a
production OOM incident, and this report does not claim one.

Environment

  • Surface: Unsloth Studio, llama.cpp backend
  • Source: unslothai/unsloth main at 45a371ce0b02fc7eeb47786c285659672a2b53ea
  • OS: macOS 26.6.2
  • Reproducer: lightweight Python child processes; no GPU or model download required

Steps to reproduce

  1. Instantiate LlamaCppBackend with a saved GgufLoadIntent.
  2. Start a lightweight Python child and terminate it with SIGKILL (observed return code
    -9).
  3. Replace load_model with a deterministic harness that records the replay, starts a
    replacement child, kills that child with SIGKILL, and reports that startup succeeded.
  4. Call the existing _respawn_if_dead() path for five independent requests.

The pre-fix result is five reloads for five requests:

replayed = [true, true, true, true, true]
load_model calls = 5

The existing one-retry-per-generation guard does not bound this sequence because the guard
is recreated for every independent request.

Expected behavior

Automatic recovery should remain available for transient deaths, but consecutive
replacement deaths without a completed generation should eventually reach an explicit
terminal state. A successful, terminal generation should restore the recovery budget for a
future independent failure.

Proposed correction

I have a focused patch that:

  • caps automatic replay at three consecutive respawns;
  • issues a one-shot recovery token tied to the exact attempt, replacement process, and
    unload epoch;
  • restores the budget only when that recovery reaches the exact data: [DONE] terminal
    event;
  • prevents a stale completion or unrelated request from resetting a newer recovery chain.

Proof collected on the patch:

  • regression tests: 5 passed;
  • adjacent Studio tests: 76 passed, 0 failed, 0 skipped;
  • real-process SIGKILL harness: 1 passed, with three reloads followed by two refusals;
  • Ruff on the two changed files: passed;
  • git diff --check: passed.

No prompts, model data, credentials, or private runtime diagnostics are included.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions