AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
fix(backend): unblock dry-run Orchestrator + Haiku 4.5 sim default (#13171)
### Why / What / How
**Why:** The Agent Simulator (dry-run) routes the Orchestrator block to
*real* execution with a platform-supplied "cheap" model. Two problems
with the pre-PR state:
1. The platform default was `"google/gemini-2.5-flash-lite"` — a string
literal that was **not a member of the `LlmModel` enum**. Since
`OrchestratorBlock.Input.model` is typed as `LlmModel`, Pydantic input
validation rejected every dry-run that hit the Orchestrator path.
2. The "cheap" model choice (Flash-Lite) was justified as
"shape-matching only", but Orchestrator dry-runs aren't shape-matching —
they pick tools and emit a finish message. A substitute that's too weak
gives an unfaithful preview of what the real run would do.
**What:**
- Register the missing `google/gemini-2.5-flash-lite` slug as a
first-class `LlmModel` member (so any deployment that prefers Flash-Lite
via `CHAT_SIMULATION_MODEL` keeps working — fixes the validation crash).
- Switch the platform default from Flash-Lite to **Claude Haiku 4.5** so
the dry-run preview better matches the real run's tool-use decisions.
- Tie both default constants (`simulator.py:_DEFAULT_SIMULATOR_MODEL`
and `copilot/config.py:simulation_model`) to
`LlmModel.CLAUDE_4_5_HAIKU.value` at import time, so any future rename
or drift breaks at module load instead of silently failing input
validation.
- Update the `TestDefaultSimulatorModel` pin to match the new default,
and add an `LlmModel(_DEFAULT_SIMULATOR_MODEL) is
LlmModel.CLAUDE_4_5_HAIKU` assertion so the bug class behind SECRT-2368
(a default that isn't an enum member) trips at unit-test time.
- Tighten the two `prepare_dry_run(OrchestratorBlock, …)` tests with the
same `LlmModel`-parseability predicate `OrchestratorBlock.Input`
enforces at runtime.
**How:**
- `backend/blocks/llm.py`: add `GEMINI_2_5_FLASH_LITE =
"google/gemini-2.5-flash-lite"` enum entry + matching `ModelMetadata`
(1M context, 64K max output, tier 1, OpenRouter).
- `backend/data/block_cost_config.py`: matching entries in `MODEL_COST`
(tier 1) and `TOKEN_COST` (`input=15, output=60` — OpenRouter's
$0.10/$0.40 × 1.5× margin × 100 credits/$, matches the existing
`GEMINI_2_0_FLASH_LITE` formula).
- `backend/executor/simulator.py`: import `LlmModel` at top, set
`_DEFAULT_SIMULATOR_MODEL = LlmModel.CLAUDE_4_5_HAIKU.value`, update the
comment to reflect the decision-quality rationale.
- `backend/copilot/config.py`: import `LlmModel` at top, set
`simulation_model` default to `LlmModel.CLAUDE_4_5_HAIKU.value`, update
the description.
- Tests: move `from backend.blocks.llm import LlmModel` to top-level per
`backend/AGENTS.md`; update the `TestDefaultSimulatorModel` pin + add an
`LlmModel`-parseability check on the default; add a parseability
assertion on the `prepare_dry_run` orchestrator tests.
- `docs/integrations/block-integrations/llm.md`: regenerated via `poetry
run python scripts/generate_block_docs.py` so `check-docs-sync` passes
with the new enum member.
**Routing note (why Haiku-via-OpenRouter is safe):** `prepare_dry_run`
injects an OpenRouter API key as the credential, and `OrchestratorBlock`
dispatches on the **credential**'s provider (`orchestrator.py:1647`),
not the model's metadata provider. So Haiku 4.5 routes through
OpenRouter's Anthropic-compat endpoint regardless of
`LlmModel.CLAUDE_4_5_HAIKU.metadata.provider == "anthropic"`. No change
to provider routing.
### Changes 🏗️
- New `LlmModel.GEMINI_2_5_FLASH_LITE` enum entry + metadata + cost
lookups.
- Dry-run simulator default flipped from Flash-Lite to Claude Haiku 4.5.
- Both default constants tied to the `LlmModel` enum at import time.
- `TestDefaultSimulatorModel` pin updated; `LlmModel`-parseability
assertion added on both the pin and the two `prepare_dry_run`
orchestrator tests.
- Regenerated `llm.md` for the new enum member.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] `poetry run pytest
backend/copilot/tools/test_dry_run.py::test_prepare_dry_run_orchestrator_block`
passes
- [x] `poetry run pytest
backend/executor/simulator_test.py::TestPrepareDryRun` passes (5 tests)
- [x] `poetry run pytest
backend/executor/simulator_test.py::TestDefaultSimulatorModel` passes (2
tests — value pin + `LlmModel` parseability)
- [x] Direct end-to-end check: `prepare_dry_run(OrchestratorBlock(),
{…})` injects `"claude-haiku-4-5-20251001"` and
`OrchestratorBlock.Input(...)` validates to `LlmModel.CLAUDE_4_5_HAIKU`
without error
- [x] Switching `CHAT_SIMULATION_MODEL=google/gemini-2.5-flash-lite`
still works (the enum entry was added for exactly this configurable
case)
- [x] `poetry run ruff format` + `poetry run ruff check` clean on all
changed files Z
Zamil Majdy committed
09368cd558ab6df09a54a370fdaae70f85ea2203
Parent: 5b38672
Committed by GitHub <noreply@github.com>
on 5/21/2026, 1:49:17 AM