feat(llm): route batch retain through a batch-capable multi-LLM member (#3649)
* feat(llm): route batch retain through a batch-capable multi-LLM member MultiLLMProvider.__getattr__ binds _provider_impl (and the OpenAI/Groq Batch-API path) to the primary member only, so a secondary member could never supply batch capacity, and HINDSIGHT_API_RETAIN_BATCH_ENABLED=true could hard-fail at startup based on the primary's capability alone. - Add supports_batch_api() and batch_provider_impl() to both LLMProvider and MultiLLMProvider. The chain version returns the first batch-capable member's impl (deterministic by declared order), so submit and a crash-recovery resume resolve to the same provider account. - MemoryEngine startup validation now checks the whole chain, not just the primary. - extract_facts_from_contents_batch_api resolves the serving impl once and targets it for the entire batch lifecycle; the request body now carries the batch member's model/provider/service_tier. * fix(llm): finish the batch-member routing — docs, contract and real tests Review follow-up on the batch-capable member selection. The behaviour was right but three files still told operators the opposite: configuration.md stated "Batch retain runs on the primary member only", and multi_llm.py's own module docstring and __getattr__ comment said the same. All three now describe what the code does — batch runs on the first batch-capable member in declared order and, once selected, does not fail over. batch_provider_impl() returned the primary's impl when nothing could serve a batch, which forced the call site to re-ask the capability question it had just answered. It now returns None as the single "cannot serve one" answer, typed as LLMInterface | None instead of Any, and resolves members through their public batch surface rather than reaching into member._provider_impl. The request body reads batch_impl.provider / .openai_service_tier directly again: provider is set by every LLMInterface and the short-circuit keeps impls without a tier from touching the attribute, so a renamed field fails loudly instead of silently dropping flex pricing. A resume now checks the batch_provider recorded at submit against the member selected now. Selection is deterministic only while the chain configuration is unchanged; edit the chain mid-flight and the old code would poll an account that has never heard of that batch_id, hanging until the wall clock ran out. Tests: the startup check moved out of the verify_llm closure into validate_retain_batch_support() so it can be called, and test_batch_api_validation.py now drives it with real providers — every test in that file previously re-implemented the check inline and asserted against its own copy. New coverage for the lifecycle running entirely on the secondary, the request body carrying that member's settings, and both resume paths. The mock_llm_config fixture honours the None contract; as a static AsyncMock it had made test_batch_api_raises_for_unsupported_provider pass vacuously. --------- Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
E
Everton Souza committed
7f7d9c13385c232c433fb9c4719026bbb4f2dde0
Parent: 2bba17e
Committed by GitHub <noreply@github.com>
on 8/20/2026, 8:20:37 AM