SIGN IN SIGN UP

fix(cache): say what the stored cache actually does, and bench it that way

Three places still described the cache as quantized after the SSD-first
overhaul made it full precision for every family.

1. The serve log announced "KV cache auto mode: TurboQuant enabled for
   compatible models; stored prefix cache quantization=none". Two different
   levers in one sentence, the misleading half first. Reading that line cost a
   whole investigation before the on-disk dtypes settled it: F16 keys and
   values, zero scale or bias tensors, nothing quantized anywhere in an 11GB
   live cache. It now leads with the cache and names the live working KV as the
   only thing TurboQuant touches.

2. bench defaulted stored quantization to q4 while serve used none, so any
   benchmark measured a codec the product no longer writes. The whole point of
   benching is to predict what users will see. Now identical to serve, with the
   same environment override for diagnostics.

3. The stored-cache selector still offered "Auto (engine-selected:
   native/TurboQuant + stored fallback)". Auto has meant full precision for
   every family since the overhaul, so the label advertised behaviour that no
   longer exists. Relabelled across all five locales.

Auto stays the default rather than an explicit none, deliberately: passing
--kv-cache-quantization with any value also sets VMLX_DISABLE_TQ_KV, which
would switch off the calibrated live in-memory TurboQuant and raise resident
memory. That is not what turning off stored quantization was for.

q8 and q4 remain selectable wherever the architecture tolerates a lossy stored
codec; families that need exact stored KV keep offering only auto and none.
J
Jinho Jang committed
6c7f50c17456c9176e8d9b3fbc6278bed105212c
Parent: 2c931e3