Skip to content

Commit a105c44

Browse files
committed
studio: exempt offers of help, and add a measured accuracy floor
Offering to help hands control back exactly like the existing "let me know" exemption. On a corpus of real model turns, "I'll do my best to help" and "allow me to assist" close a clarification request and never precede a tool call, but they were read as intent and re-prompted. "help you" keeps its plan reading when an action verb follows it. The new test scores the classifier against 300 turns captured from three local GGUF models, each one a finished answer: the turn called no tool, and three regenerations behind the production nudge produced no tool call either. Over those turns, wasted nudges go from 36 (12.0%) on main to 5 (1.7%), and retries whose text would be discarded from 60 (20.2%) to 1 (0.3%). Until now these patterns were tuned on hand-written example sentences, which cannot show how often the classifier is right on real output.
1 parent 20fe689 commit a105c44

3 files changed

Lines changed: 405 additions & 2 deletions

File tree

studio/backend/core/inference/tool_call_parser.py

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -173,14 +173,23 @@
173173
r"|review|inspect|read|gather|examine|retrieve|browse|consult|verify"
174174
r"|confirm|compute|calculate|determine|identify|render)"
175175
)
176+
# Offering to help hands control back exactly like "let me know": measured on real
177+
# turns, "I'll do my best to help" and "allow me to assist" close a clarification
178+
# request and never precede a tool call. "help you" keeps its plan reading when an
179+
# action follows it ("I'll help you search the web").
180+
_HELP_OFFER = (
181+
r"(?:do(?:ing)?\s+my\s+best|try\s+my\s+best|be\s+(?:able|happy|glad)\s+to\b"
182+
r"|assist\b|help\s+you\b(?!\s+" + _ACTION_VERB + r")|give\s+you\s+accurate\b)"
183+
)
176184
# Forward-looking intent: the model says what it *will* do, not a final answer.
177185
INTENT_SIGNAL = re.compile(
178186
r"(?im)("
179187
# Direct intent ("I'll"); lookahead drops negated forms ("I will not").
180-
r"\b(i['\u2019](ll|m going to|m gonna)|i am (going to|gonna)|i will|i shall)\b(?!\s+(?:not|never)\b)"
188+
r"\b(i['\u2019](ll|m going to|m gonna)|i am (going to|gonna)|i will|i shall)\b"
189+
r"(?!\s+(?:not|never)\b)(?!\s+" + _HELP_OFFER + r")"
181190
r"|"
182191
# "let me know" hands control back rather than announcing an action.
183-
r"\b(?:let me|allow me)\b(?!\s+(?:not|never|know)\b)"
192+
r"\b(?:let me|allow me)\b(?!\s+(?:not|never|know)\b)(?!\s+to\s+" + _HELP_OFFER + r")"
184193
r"|"
185194
# Step/plan framing. "first" must open a sentence and be followed by a plan
186195
# (pronoun, "my/our plan", or an action verb); otherwise it is prose ("The

0 commit comments

Comments
 (0)