SIGN IN SIGN UP

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

0 0 83 Python

fix(backend/copilot): mandate find_block before refusal + suppress false empty-completion on retry (#13086)

## Why

Two production-visible regressions surfaced in the same dev session
(`d077d025-f6a3-455f-836f-9fe3039b2b98`):

1. **AutoPilot refused without searching.** User asked "Post a test post
to
   linkedin for me". The model emitted *"We don't have a native LinkedIn
integration yet. LinkedIn's API is also heavily restricted…"* with
**zero
tool calls** — never invoked `find_block`. When the user pushed back,
the
   model immediately found `PostToLinkedInBlock` (Ayrshare) on the first
   `find_block(query="linkedin post")`. The model's own thinking from
   langfuse: *"The user is correct — I jumped straight to capability gap
without first searching for blocks. Per my instructions, I should always
`find_block` first for any integration request."* — i.e. the rule
existed
   but was overridden by a vivid worked example in the system prompt.

2. **"Model returned an empty response" overlay on top of working
output.**
   In the same session's recovery turn, after `find_block` + `run_block`
succeeded and the model emitted *"Yes, we have a `PostToLinkedInBlock`
   via Ayrshare — and you're already connected…"*, a retry-adapter
   recreated mid-turn lost the prior attempt's `has_started_text` /
   `_any_tool_results_seen` state and false-fired
   `code=empty_completion` on the next ResultMessage.

Root causes:

- The Langfuse "CoPilot Prompt" Section 4 ("Capability Check") + Example
3
  (Shopify) explicitly modelled "We don't have a native [Platform]
  integration yet" as the *correct* response when the model "knows" the
  platform isn't supported — training the model to refuse without
searching. The shared tool-notes supplement said "find_block first" but
  the worked example won.
- `SDKResponseAdapter` is rebuilt fresh on context-reduce retries
  (service.py:4510). Its empty-completion guard relies on per-attempt
  state, so a benign empty trailing ResultMessage on a retry — when the
  prior attempt already streamed text/tools to the wire — is
  indistinguishable from a true SECRT-2252 ghost finish and surfaces a
  spurious error overlay.

## What

Code-side fixes (Langfuse prompt is also patched on the `latest`/dev
label out-of-band; prod stays on its existing version pending review):

- **`prompting.py`** — rewrite `SHARED_TOOL_NOTES` "Tool Discovery
Priority" to mark `find_block` as MANDATORY before any "no integration"
  reply. Add an explicit "Anti-pattern: refusing without searching"
  section that enumerates the exact phrases (`We don't have a native X
  integration yet.`, `There's no block for X.`, etc.) and shows the
  correct 3-step flow.
- **`response_adapter.py`** — add
  `prior_attempt_emitted_visible_content: bool = False` to
  `SDKResponseAdapter`; `_is_empty_completion` returns False when the
  flag is set, so the SECRT-2252 guard only fires when the *broader
  stream* has produced no content, not just the current attempt.
- **`service.py`** retry path — before swapping the adapter on
  `_reduce_context` retry, set the new adapter's
  `prior_attempt_emitted_visible_content` based on the prior adapter's
  `has_started_text` / `has_started_reasoning` /
  `_any_tool_results_seen` (and forward the flag transitively across
  multi-retry chains).
- **Tests** — `prompting_test.py::TestToolDiscoveryPriorityAntiPattern`
  (3 assertions on the new mandatory language + anti-pattern phrases +
  correct-flow template); `response_adapter_test.py` adds

`test_retry_adapter_with_prior_emitted_content_suppresses_empty_completion`
  (positive: empty success on retry → StreamFinish, no StreamError) and

`test_retry_adapter_without_prior_content_still_surfaces_empty_completion`
  (counter-test: SECRT-2252 still fires when the flag is False).

## How

The three fixes layer:

1. **System-prompt rule** (defence in depth) — code-side supplement now
   explicitly forbids the failure-mode phrases by exact wording; the
   model can pattern-match its own draft and reject it.
2. **Langfuse worked example** (root cause) — Section 4 + Example 3 of
   "CoPilot Prompt" rewritten on `latest` v30 to model search-first
   behaviour. Prod (`production` v27) untouched until reviewed.
3. **Adapter false-positive guard** — surface "model returned an empty
   response" only when the *whole stream* produced nothing, not just the
   current retry attempt.

## Test plan

- [x] `pytest backend/copilot/sdk/response_adapter_test.py -k
"empty_completion or prior_attempt or retry_adapter"` — 3 passed.
- [x] `pytest
backend/copilot/sdk/response_adapter_test.py::test_retry_adapter_with_prior_emitted_content_suppresses_empty_completion
backend/copilot/sdk/response_adapter_test.py::test_retry_adapter_without_prior_content_still_surfaces_empty_completion`
— 2 passed.
- [x] Direct supplement assertion (`from backend.copilot.prompting
import get_sdk_supplement; ...`) — all pass.
- [x] `ruff check` clean on edited files.
- [x] Validated against trace `9f3feb3dd843aa1e537de92f23666cc9` /
      `746b49e0f6012ff94035d0ceb3f7fca6` (the LinkedIn dev session).
- [ ] Manual dev verification with a fresh integration request after
      `latest` v30 propagates (≤5 min Langfuse cache TTL).
Z
Zamil Majdy committed
8d9ecb8197c39ecf61b760cd60f930099d7b294a
Parent: 12cc0f6
Committed by GitHub <noreply@github.com> on 5/12/2026, 9:49:45 AM