AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
fix(backend/copilot): suppress empty-completion overlay false positives (orphan-flush + specific error subtypes) (#13090)
## Why
After PR #13086 deployed (retry-recreate fix), **6 distinct dev
sessions** in the last 48h still hit the "The model returned an empty
response" overlay (per gcloud + langfuse forensics). Same user-visible
symptom, **two distinct root causes** the prior PR didn't cover. This PR
fixes both.
| Category | Sessions | Root cause | Fixed by |
|---|---|---|---|
| **A** | b0e8ca83, 48a65d73, deb7cd95 | Sticky `_any_orphan_flush_seen`
flag + empty trailing AssistantMessage on otherwise-healthy turn →
false-positive overlay on top of working tool widgets | This PR |
| **B** | d077d025, 1f1f65a3 | Pre-PR-#13086 retry-recreate ghost-finish
| PR #13086 (already shipped) |
| **C** | 835da550 | `error_max_budget_usd` ResultMessage subtype
shadowed by the empty-completion overlay; user sees "empty response"
instead of "budget exceeded" | This PR |
**Regression origin:** PR #13052 (merged 2026-05-08, 4 days before this
PR) introduced the orphan-flush guard (Path B). It correctly fixed one
specific Kimi K2.6 failure but the discriminator was too coarse —
`_any_orphan_flush_seen + _last_assistant_had_empty_content` fires on
healthy long turns whenever one earlier orphan flush happened. The
PostToolUse stash race made orphan flushes routine, so this
false-positive started showing up immediately.
## 5-whys (Category A)
1. **Why error overlay?** `_should_surface_empty_completion` path B
(`_any_orphan_flush_seen AND _last_assistant_had_empty_content`)
returned True.
2. **Why orphan-flush sticky?** `_any_orphan_flush_seen` flips True for
any flush call with unresolved tools — sticky for the whole turn.
3. **Why does that conflate?** The flush path has two outcomes — real
output recovered from stash (benign) and empty fallback because stash
was empty (the actual signal). The flag treated them identically.
4. **Why empty trailing AssistantMessage?** Long turns naturally end
with empty trailing emissions (cancellation, thinking-only edge cases).
Combined with sticky orphan-flush flag, path B fires on any one earlier
orphan-flush.
5. **Why doesn't the guard ask "did the user see real content this
turn?"** It only inspected the current message + a turn-wide
orphan-flush boolean, never cross-checking against what was actually
emitted to the wire.
## What
### Category A fix — real-content discriminator
- **`_any_real_tool_result_seen`** (new): flips True only on
- `UserMessage` `ToolResultBlock` (real tool result via SDK)
- `flush_unresolved_tool_calls` when `pop_pending_tool_output` returned
non-None (real recovered output)
- **NOT** on the empty-fallback flush branch
- **`emitted_real_content_to_wire`** (new property): `has_started_text
OR _any_real_tool_result_seen OR prior_attempt_emitted_visible_content`.
Deliberately excludes `has_started_reasoning` — a thinking-only turn
without text or real tool output IS the failure mode the guard is meant
to catch.
- **`_should_surface_empty_completion`** early-returns False when
`emitted_real_content_to_wire=True`. The existing ghost-finish (path A)
and orphan-only-empty-fallback (path B) still fire as designed.
- **Retry-recreate forwarding** (service.py): forwards
`prior_adapter.emitted_real_content_to_wire` (the strict property), so a
prior reasoning-only attempt does not suppress the guard on the
recreated adapter.
### Category C fix — specific error subtypes bypass the overlay
- New `_SPECIFIC_ERROR_SUBTYPES = {"error_max_budget_usd",
"error_max_turns"}` constant.
- `_should_surface_empty_completion` returns False for these subtypes
(they have their own user-facing reason and must not be shadowed).
- ResultMessage branch routes them to specific `StreamError` codes:
- `error_max_budget_usd` → `code="max_budget_exhausted"`, *"The turn
ended because it exceeded the budget…"*
- `error_max_turns` → `code="max_turns_exhausted"`, *"…it reached the
maximum number of LLM calls…"*
### Local dev tool — `scripts/replay_session_trace.py`
Standalone script (same style as `scripts/download_transcripts.py`) that
pulls a langfuse trace by session ID, reconstructs the SDK message
stream, runs it through a fresh adapter, and prints whether
`StreamError(code="empty_completion")` would fire. Used to validate this
PR against the 3 reported failure sessions.
Run as:
```bash
LANGFUSE_PUBLIC_KEY=… LANGFUSE_SECRET_KEY=… LANGFUSE_HOST=… \
poetry run python -m scripts.replay_session_trace <session_id> [--subtype <subtype>]
```
## Test plan
- [x] **Synthetic regressions** (committed):
- `test_empty_completion_suppressed_when_real_tool_result_already_seen`
— Category A reproducer
-
`test_empty_completion_still_fires_on_orphan_with_only_empty_fallbacks`
— counter-test (orphan guard still fires when no real content)
- `test_empty_completion_suppressed_when_text_already_streamed` —
text-already-streamed counter
- `test_real_tool_result_flag_set_by_stashed_flush` — flag setter
- `test_error_max_budget_usd_surfaces_specific_error_not_empty_overlay`
— Category C
- `test_error_max_turns_surfaces_specific_error` — Category C
- [x] **Real-trace replay** via `scripts/replay_session_trace.py`
against all 3 reported sessions:
- `b0e8ca83` (Category A) → `stream_errors: []` (was firing
`empty_completion`)
- `1f1f65a3` (Category A) → `stream_errors: []`
- `835da550` (Category C) → `stream_errors: [{code:
"max_budget_exhausted", …}]` (was firing `empty_completion`)
- [x] Full `response_adapter_test.py` — **73 passed** locally.
- [x] `ruff check` clean on edited files.
- [x] CodeRabbit review addressed: strict forwarding, no-suppressors,
tool-result match by (name, input).
- [ ] Manual dev verification after rollout against the failing session
pattern. Z
Zamil Majdy committed
04ac81c17b4ac05cbb0c849e92a61eaf46e1f736
Parent: aa4d94a
Committed by GitHub <noreply@github.com>
on 5/12/2026, 1:00:18 PM