fix(llm): recover empty tool-loop responses instead of returning null#929
fix(llm): recover empty tool-loop responses instead of returning null#929leo922oel wants to merge 1 commit into
Conversation
Walkthrough
ChangesEmpty response recovery
Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant execute_tool_loop
participant honcho_llm_call_inner
participant synthesis
execute_tool_loop->>honcho_llm_call_inner: request tool-enabled response
honcho_llm_call_inner-->>execute_tool_loop: empty tool-less response
execute_tool_loop->>synthesis: select empty-response recovery prompt
synthesis-->>execute_tool_loop: recovered final response
Possibly related issues
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/llm/tool_loop.py`:
- Around line 499-526: Update the empty-response handling in
src/llm/tool_loop.py lines 499-526 to treat None and blank strings as empty,
route every non-retriable empty response to forced synthesis regardless of
empty_response_retries, and increment iteration before breaking so the synthesis
call is counted. Add deterministic coverage in
tests/llm/test_dialectic_minimal_empty_recovery.py lines 61-83 for None and two
consecutive empty responses, asserting synthesis recovery and result.iterations
== 2.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3f86c1f1-8f8a-4632-bcd0-2552cca4f0e5
📒 Files selected for processing (2)
src/llm/tool_loop.pytests/llm/test_dialectic_minimal_empty_recovery.py
When the model returns an empty response with no tool calls, the tool loop's empty-response retry only fires when iterations remain (`iteration < max_tool_iterations - 1`). At the dialectic `minimal` tier (MAX_TOOL_ITERATIONS=1) that guard is `0 < 0 == False`, so the retry can never fire and the tool-less branch returns empty content without reaching the post-loop synthesis call. The empty string then surfaces as a null dialectic response (the chat endpoint maps a falsy answer to None), so broad `minimal` queries returned null even though the same query answered correctly at `low`/`medium`. Decouple empty-response recovery from the iteration budget: try one in-loop nudge while a retry and an iteration remain, otherwise break to the forced synthesis call so a final answer is always produced. This applies whenever a response cannot be retried, including a repeat empty after the nudge; a None completion is treated as empty; and the synthesis call is counted in the reported iteration total. General tool-loop correctness fix; `minimal` is where it is guaranteed to surface. The synthesis prompt and log distinguish empty-recovery from genuine max-iteration exhaustion, and the streaming path is unchanged. Adds deterministic regression tests covering empty-string and None recovery at a single iteration, a repeat empty routed to synthesis, and the in-budget nudge control.
0b31359 to
dd989ab
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/llm/tool_loop.py (1)
518-518: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winWrap the nudge text to the configured line limit.
Line 518 exceeds 88 characters including indentation.
Proposed fix
- "Your last response was empty. Provide a concise answer " + "Your last response was empty. " + "Provide a concise answer "As per coding guidelines, “keep lines within 88 characters to match Black-compatible formatting.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/llm/tool_loop.py` at line 518, Wrap the nudge text string in the tool-loop response handling near the empty-response message so every source line, including indentation, stays within the configured 88-character limit while preserving the exact message content.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/llm/tool_loop.py`:
- Line 518: Wrap the nudge text string in the tool-loop response handling near
the empty-response message so every source line, including indentation, stays
within the configured 88-character limit while preserving the exact message
content.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: eab7da91-5459-4206-b9f0-30cfd837ddde
📒 Files selected for processing (2)
src/llm/tool_loop.pytests/llm/test_dialectic_minimal_empty_recovery.py
|
Finn-loop review of dd989ab CI: not configured (only CodeRabbit runs; no required lint/test/typecheck checks on this branch) ReviewPR #929 fixes a bug where 1. Must fix before merge
2. Should fix soonNone. The code changes are well-scoped:
3. Safe to mergeNo — CI gate is missing. Automated review of the code itself finds no defects, but the lack of required CI checks means a human must verify that linting, type-checking, and tests pass before merging. NoteThis reviewer's account does not have admin rights to this repository, so labels ( |
2 participants
Description
peer.chat(..., reasoning_level="minimal")returnednullon broad queries the model answered without a tool call. The tool loop's empty-response retry is gated oniteration < max_tool_iterations - 1, which is0 < 0 == Falsefor theminimaltier (MAX_TOOL_ITERATIONS=1), so the retry never fired and the tool-less branch returned empty content without reaching the post-loop synthesis call. The empty string then surfaced asnull(the chat endpoint maps a falsy answer toNone).lowandmediumwere unaffected because they allow>= 2iterations.Motivation
Broad
minimalqueries silently returnednulleven though the same query answered correctly atlow/medium, so callers relying onminimalreceived no memory and no error signal.Changes
src/llm/tool_loop.py: decouple empty-response recovery from the iteration budget. Keep the nudge-and-loop when iterations remain, otherwise break to the forced synthesis call so a final answer is always produced. This is a general tool-loop correctness fix;minimalis simply where it is guaranteed to surface. The synthesis prompt and log now distinguish empty-recovery from genuine max-iteration exhaustion, and the streaming path is left unchanged (not stream_final).tests/llm/test_dialectic_minimal_empty_recovery.py: deterministic regression test (mocked inner LLM) covering the single-iteration recovery path and the multi-iteration control.Testing
tests/llm/full suite: 242 passed (new regression test RED to GREEN).ruff check/ruff format/basedpyrightclean.Summary by CodeRabbit
Bug Fixes
Tests
Nonefinal responses, including single-iteration forced synthesis and multi-iteration nudge/retry behavior.