SIGN IN SIGN UP

fix(retain): batch a document's chunk embeddings instead of one request per chunk (#3785)

In `chunks` extraction mode a retain issued one embedding request per chunk. The
streaming producer fans out one task per chunk and each task embeds only its own
chunk's facts, so `generate_embeddings_batch` — which already takes a list — was
never handed more than one text on that path. That shape is right for the
extraction path (every chunk needs its own LLM call and the fan-out lets those
overlap), but in chunks mode extraction is a no-op, so the single round trip is
the entire per-chunk cost and ingest ends up bounded by it (#3784).

Keep the fan-out and move the batching one layer down: `CoalescingEmbedder`
merges whatever embedding calls are concurrent into one backend request, sized
from the backend's own per-request limit. Nothing waits on a timer — a caller
that arrives while a backend slot is free is dispatched on the next event-loop
tick — so a lone caller pays no added latency while a fan-out naturally forms
full batches. One coalescer per retain: the backends read the ambient bank id
for cost attribution, so texts from different banks never share a request.

Also makes the TEI per-request batch size configurable
(`HINDSIGHT_API_EMBEDDINGS_TEI_BATCH_SIZE`, default 32, previously hardcoded);
it is what the coalescer sizes its batches from, so it tunes both.

Measured on the producer fan-out with 300 chunks against a TEI stub: 300
requests -> 10, and 861 -> 8636 chunks/s at 2ms per request, 381 -> 2413
chunks/s at 30ms. End to end on a 600k-char document: 415 embed requests -> 19.

Claude-Session: https://claude.ai/code/session_01HVZ94d143NPSsZjbnvwqba
N
Nicolò Boschi committed
ca2c983e1aa639bb0b5800b402874866f8d31439
Parent: cea8cab
Committed by GitHub <noreply@github.com> on 8/25/2026, 11:40:42 AM