SIGN IN SIGN UP

feat(llm): route batch retain through a batch-capable multi-LLM member (#3649)

* feat(llm): route batch retain through a batch-capable multi-LLM member

MultiLLMProvider.__getattr__ binds _provider_impl (and the OpenAI/Groq
Batch-API path) to the primary member only, so a secondary member could
never supply batch capacity, and HINDSIGHT_API_RETAIN_BATCH_ENABLED=true
could hard-fail at startup based on the primary's capability alone.

- Add supports_batch_api() and batch_provider_impl() to both LLMProvider and
  MultiLLMProvider. The chain version returns the first batch-capable
  member's impl (deterministic by declared order), so submit and a
  crash-recovery resume resolve to the same provider account.
- MemoryEngine startup validation now checks the whole chain, not just the
  primary.
- extract_facts_from_contents_batch_api resolves the serving impl once and
  targets it for the entire batch lifecycle; the request body now carries the
  batch member's model/provider/service_tier.

* fix(llm): finish the batch-member routing — docs, contract and real tests

Review follow-up on the batch-capable member selection.

The behaviour was right but three files still told operators the opposite:
configuration.md stated "Batch retain runs on the primary member only", and
multi_llm.py's own module docstring and __getattr__ comment said the same. All
three now describe what the code does — batch runs on the first batch-capable
member in declared order and, once selected, does not fail over.

batch_provider_impl() returned the primary's impl when nothing could serve a
batch, which forced the call site to re-ask the capability question it had just
answered. It now returns None as the single "cannot serve one" answer, typed as
LLMInterface | None instead of Any, and resolves members through their public
batch surface rather than reaching into member._provider_impl.

The request body reads batch_impl.provider / .openai_service_tier directly
again: provider is set by every LLMInterface and the short-circuit keeps impls
without a tier from touching the attribute, so a renamed field fails loudly
instead of silently dropping flex pricing.

A resume now checks the batch_provider recorded at submit against the member
selected now. Selection is deterministic only while the chain configuration is
unchanged; edit the chain mid-flight and the old code would poll an account
that has never heard of that batch_id, hanging until the wall clock ran out.

Tests: the startup check moved out of the verify_llm closure into
validate_retain_batch_support() so it can be called, and
test_batch_api_validation.py now drives it with real providers — every test in
that file previously re-implemented the check inline and asserted against its
own copy. New coverage for the lifecycle running entirely on the secondary, the
request body carrying that member's settings, and both resume paths. The
mock_llm_config fixture honours the None contract; as a static AsyncMock it had
made test_batch_api_raises_for_unsupported_provider pass vacuously.

---------

Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
E
Everton Souza committed
7f7d9c13385c232c433fb9c4719026bbb4f2dde0
Parent: 2bba17e
Committed by GitHub <noreply@github.com> on 8/20/2026, 8:20:37 AM