SIGN IN SIGN UP

fix(llamacpp): report a missing llama.cpp instead of endless connection errors (#3758)

Running the published Docker image with HINDSIGHT_API_LLM_PROVIDER=llamacpp
downloaded a 3.5 GB model, crash-looped, and — once a model was supplied by
hand — failed every retain and reflect with an unexplained APIConnectionError
against 127.0.0.1 (#3733). Three separate defects stacked up:

* A failed start still installed the shared server: it was assigned to the
  module global before start() was awaited, so every later call took the
  "already started" branch and built an OpenAI client against a port nothing
  listens on. The real error (no module named 'llama_cpp') surfaced once, at
  boot, where LLM verification only warns — and was masked from then on. The
  server is now published only once it is serving, and a subprocess left
  behind by a timed-out start is reaped so a retry cannot stack another one.

* The model downloaded before anything checked whether a server could run at
  all. The `local-llm` extra is now required up front, with a message naming
  both ways out: the extra for local installs, the llama.cpp sidecar for the
  published image, which deliberately omits it.

* HINDSIGHT_API_MODEL_INIT_TIMEOUT — the documented knob for a slow first-time
  download — had no effect in Docker, because the container entrypoint waits
  for /health on its own undocumented timer and killed the container at 300s
  regardless. That wait now follows the API's cap when it is the longer of the
  two, plus a grace period so the API reports its own timeout first.

Docs: the configuration page advertised the built-in provider with no Docker
caveat and the image variants table read as "works out of the box except the
LLM", so the setup looked supported. Both now point at the sidecar compose
file, HINDSIGHT_API_STARTUP_WAIT_SECONDS is documented, and auto-downloaded
models are noted as needing persistent storage.
N
Nicolò Boschi committed
bdee2bc88db6de2af44dff434181969633527ee5
Parent: 3f14672
Committed by GitHub <noreply@github.com> on 8/24/2026, 11:00:50 AM