fix(llamacpp): report a missing llama.cpp instead of endless connection errors (#3758)
Running the published Docker image with HINDSIGHT_API_LLM_PROVIDER=llamacpp downloaded a 3.5 GB model, crash-looped, and — once a model was supplied by hand — failed every retain and reflect with an unexplained APIConnectionError against 127.0.0.1 (#3733). Three separate defects stacked up: * A failed start still installed the shared server: it was assigned to the module global before start() was awaited, so every later call took the "already started" branch and built an OpenAI client against a port nothing listens on. The real error (no module named 'llama_cpp') surfaced once, at boot, where LLM verification only warns — and was masked from then on. The server is now published only once it is serving, and a subprocess left behind by a timed-out start is reaped so a retry cannot stack another one. * The model downloaded before anything checked whether a server could run at all. The `local-llm` extra is now required up front, with a message naming both ways out: the extra for local installs, the llama.cpp sidecar for the published image, which deliberately omits it. * HINDSIGHT_API_MODEL_INIT_TIMEOUT — the documented knob for a slow first-time download — had no effect in Docker, because the container entrypoint waits for /health on its own undocumented timer and killed the container at 300s regardless. That wait now follows the API's cap when it is the longer of the two, plus a grace period so the API reports its own timeout first. Docs: the configuration page advertised the built-in provider with no Docker caveat and the image variants table read as "works out of the box except the LLM", so the setup looked supported. Both now point at the sidecar compose file, HINDSIGHT_API_STARTUP_WAIT_SECONDS is documented, and auto-downloaded models are noted as needing persistent storage.
N
Nicolò Boschi committed
bdee2bc88db6de2af44dff434181969633527ee5
Parent: 3f14672
Committed by GitHub <noreply@github.com>
on 8/24/2026, 11:00:50 AM