SIGN IN SIGN UP

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

0 0 80 Python

feat(backend/copilot): switch main client between OpenRouter and Anthropic-direct (#13034)

## Why

Today copilot routes every LLM call through OpenRouter. We want the SDK
+ baseline paths to talk directly to Anthropic (api.anthropic.com) so we
avoid the OR proxy hop for models we run on Anthropic anyway, while
keeping non-Anthropic helpers (title generation, future builder helpers)
on OR — they need providers Anthropic can't serve.

The plumbing for the SDK side mostly existed (the SDK env builder
already had a direct-Anthropic mode), but the baseline path and the
title client both passed through the same OpenRouter-pinned
`AsyncOpenAI` singleton, so flipping `CHAT_USE_OPENROUTER=false` would
silently break aux calls.

## What

- **Default flipped** — `CHAT_USE_OPENROUTER` now defaults to `false`
(direct-Anthropic main path). OpenRouter is opt-in via env var.
- **Config split** — added `aux_api_key` / `aux_base_url` /
`direct_anthropic_api_key` fields to `ChatConfig`, plus
`main_client_credentials` / `aux_client_credentials` /
`aux_uses_openrouter` / `aux_provider_label` / `openrouter_active` /
`effective_transport` properties so call sites read intent instead of
re-deriving URL checks. URL host comparison uses `_host_matches()`
(urlparse-based, fixes a CodeQL incomplete-substring sanitization).
- **Two client functions** — `_get_main_client()` (transport-aware:
OpenRouter or `https://api.anthropic.com/v1/`) and `_get_aux_client()`
(defaults to OR; falls back to main creds for single-key deployments).
`_get_openai_client` kept as a back-compat alias.
- **Boot validators** — `_validate_sdk_model_vendor_compatibility`
(non-Anthropic slug + direct mode → boot fail) and
`_validate_aux_client_for_direct_main` (catches `aux_base_url` without
key, subscription-mode + no aux key + no direct key, non-Anthropic title
model paired with Anthropic-pointed aux client).
- **Title generation** uses the aux client; provider label resolves to
`open_router` / `anthropic` / `openai` via `aux_provider_label`. Title
model is normalized for the Anthropic OpenAI-compat endpoint (strip
`anthropic/` prefix and dots) when aux lands on Anthropic.
- **Baseline path** normalizes the model slug for the active transport
(`model_normalize.py`), drops OR-only `usage.include`, `reasoning`, and
`stream_options` (Anthropic compat 400s on these) when not on OR, swaps
to native `thinking` extra_body in direct mode, and clamps
`budget_tokens` + `max_tokens` against the model's published max output
(Anthropic compat requires `max_tokens > budget_tokens`).
- **Anthropic rate card** — vendored `anthropic_rates.json`
(LiteLLM-derived, refreshed weekly via cron).
`compute_anthropic_cost_usd()` computes USD when OR's `usage.cost`
extension is absent (direct mode), with TTL-aware cache-write
multipliers (1.25× input for 5m, 2.0× for 1h) and a 0.1× cache-read
multiplier. Cache token buckets subtracted from `prompt_tokens` to keep
the three buckets disjoint. Falls back to opus-4-1 rates for unknown
Anthropic slugs (over-bill rather than mis-bill). Drops `litellm`
dependency entirely (snyk fix — repeated auth-bypass / password-hash
CVEs).
- **Cost recovery safety net** — when the upstream stream finishes
without a `usage` chunk in direct mode (truncated SSE / mid-stream
disconnect), recover cost from the rate card with tiktoken-estimated
tokens so the rate-limit charge can't be bypassed. OR mode skips
recovery (OR markup ≠ raw Anthropic pricing).
- **Shared normalizer** — `_normalize_model_name` extracted into
`copilot/model_normalize.py` so SDK and baseline share one rule. SDK
keeps the symbol as a thin wrapper passing its own `config` reference,
so existing monkeypatch fixtures keep driving the decision.
- **Shared helper** — `_extract_cache_creation_tokens` lives in
`token_tracking.py` and is consumed by both `service.py` and
`baseline/service.py`.

## How

Switching is a deploy-wide env flip:

```bash
# Direct Anthropic (new default)
CHAT_USE_OPENROUTER=false
ANTHROPIC_API_KEY=sk-ant-...
# (optional — defaults to OPEN_ROUTER_API_KEY for aux client)
CHAT_AUX_API_KEY=sk-or-v1-...

# OpenRouter (opt-in)
CHAT_USE_OPENROUTER=true
```

No LaunchDarkly flag — the existing `CHAT_USE_OPENROUTER` env var is
enough for a deployment-wide cutover. Per-tenant rollout can layer LD on
top later if needed.

USD cost tracking is preserved: SDK direct mode uses the CLI's
`total_cost_usd` (rate-card accurate for Anthropic models); baseline
direct mode uses the rate-card lookup, recorded with
`provider="anthropic"`. Aux calls keep using OR's `usage.cost` extension
when on OR; when aux falls back to Anthropic, cost is computed via the
rate card at the title-cost recorder.

### Test plan

- [x] `pytest
backend/copilot/{config,anthropic_rate_card,model_normalize,service_unit,baseline/service_unit}_test.py`
+ `sdk/{service,service_helpers,openrouter_cost,env}_test.py` — all
green locally
- [x] New tests cover stream_options gating, direct-mode provider label,
cost recovery on missing usage chunk
- [x] Smoke verified `ChatConfig.main_client_credentials` returns
`(or-key, or-url)` in OR mode and `(anthropic-key,
https://api.anthropic.com/v1/)` in direct mode
- [x] /pr-test E2E on native stack — both transports, both modes,
paywall resumption, title generation, thinking-budget clamping, stream
truncation cost recovery
- [x] Cost validation on all 4 combos (direct×baseline, direct×thinking,
OR×baseline, OR×thinking) — `cost_usd` correctly persisted to
`PlatformCostLog` with `trackingType=cost_usd` (provider-reported, not
estimated)
Z
Zamil Majdy committed
1ec097cbd986f023ebf0dc5530279a4b36d94b8f
Parent: f360ada
Committed by GitHub <noreply@github.com> on 5/9/2026, 2:13:32 AM