studio: revisit default-on tool-call nudging, starting with the external loop

[danielhanchen]

Opening this as a decision to make rather than a bug. Measurements below, all reproducible.

What nudging is

When a tool-enabled model answers with a plan instead of calling a tool ("I'll search for that now"), the loop re-prompts it to act. Gate: is_short_intent_without_action / INTENT_SIGNAL in studio/backend/core/inference/tool_call_parser.py. Born in #4769.

Every firing I can measure is a false positive

tests/data/plan_vs_answer.jsonl is 300 real turns, all of them finished answers by construction. The classifier fires on 4 of them, and all 4 are the model describing its own methodology:

[qwen3-0.6b/format] 'What is your plan for answering a question you are unsure about?'
   "When unsure about a question, I plan to first search the web for the information
    using the web_search function..."

That is not a stall. Nudging it costs a whole extra generation.

I then captured 624 fresh turns through llama-server with the real Studio tool catalog, the real system prompt from _build_tool_action_nudge, and the real nudge message, across Qwen3-0.6B, Qwen3-1.7B and Llama-3.2-1B. The nudge converted 52 turns into a tool call. The classifier fired on none of them, because those turns are 75% confident hallucinated answers, 13% capability disclaimers and 12% questions back to the user. Not one is a plan-without-action announcement.

Adding the existing corpus: 924 real turns, no observed instance of the phenomenon the feature exists to catch.

I also ran #4769's own trigger category directly (creation tasks: "Create a weather dashboard for London as an HTML canvas"), with the Studio system prompt attached. 24 turns, every one acted-immediately. Zero stalls.

The likeliest explanation is that the system-prompt half already fixed it

#4769 shipped two things: system-prompt guidance, and this loop-level nudge. Its system-prompt half added "Always call tools directly. Never describe what you plan to do, just call the tool immediately." Today's prompt carries directive tool and canvas guidance in the same spirit. Under that roof the loop-level nudge may simply have nothing left to catch, which is what a classifier firing 4 times in 300 turns with no true positives looks like from the inside.

What it has cost

• Fix Studio safetensors tool reprompt reasoning leakage #7134, Studio: Don't re-prompt finished answers in the tool loop #7505 and studio: stop nudging a turn that asks the user for a missing detail #8946: three PRs narrowing false positives
• [Bug] Unsloth Studio sometimes nudges models for tool calls when they did not make any tool call #8907: the only user-facing report, and it was a false positive (a clarification turn getting nudged)
• fix(studio): gate external tool nudges and preserve retry context #9125: the external loop replaying a stall as user -> user, rewriting the user's own message up to three times, stacking up to four answers into one message and destroying prefix reuse from inside the first user turn

The measurement that decided me

A/B of the two conversation shapes on a real model, 108 generations per arm, same prompts and seeds:

BROKEN shape (user -> user)               tool calls: 85/108  (79%)
FIXED  shape (user -> assistant -> user)  tool calls: 39/108  (36%)

The broken shape extracts more than twice as many tool calls. It does so by hiding the model's own previous answer from it, so the model does not stand by what it already said. A feature whose measured benefit comes from withholding context from the model is not in good health.

Proposal

Revisit default-on for the external loop specifically (studio/backend/core/inference/studio_tool_loop.py), by not having the frontend send nudge_tool_calls: true on the external branch of chat-adapter.ts.

Scoped there because that path is the newest and the only one where firing causes corruption. Before #8665 (2026-08-13) external providers did not run this loop at all, so turning it off returns to the state of two weeks earlier and touches nothing on the GGUF and safetensors loops, where the feature has been stable since April.

Explicit nudge_tool_calls: true would still work for anyone who wants it.

What would change my mind

I have not tested, and any of these could hold the phenomenon:

• The post-tool position. llama_cpp.py keeps a separate _post_tool_reprompts counter capped at 1, which suggests someone once saw stalls after a tool had run. Every turn I captured was a first turn.
• Larger models. My sweeps topped out at 1.7B. The [Bug] Unsloth Studio sometimes nudges models for tool calls when they did not make any tool call #8907 report was Qwen3.8-27B.
• Non-English. INTENT_SIGNAL is anchored on English first-person modals, so it cannot fire at all in other languages, and I have no data on whether stalls happen there.

Reproducing

The capture harness is studio/backend/tests/tools/capture_plan_corpus.py (I will open that separately). It captures classifier-independently, so it collects turns the classifier misses rather than mirroring it, and runs a control arm so a turn only counts when the nudge specifically caused the call. --models takes any GGUF added to MODELS.

Related: #4769, #8907, #8946, #9125, #9126.