SIGN IN SIGN UP

fix(vlm): no_chunked_prefill protects vision spans, it does not forbid chunking

gemma4 sets no_chunked_prefill from use_bidirectional_attention == "vision",
and that config DEFAULTS to "vision". Honouring it as an absolute kill switch
meant gemma4 media prompts NEVER chunked, so an 80,611-token media
conversation reached

  [metal::malloc] Attempting to allocate 207,940,266,272 bytes which is
  greater than the maximum allowed buffer size of 86,586,540,032 bytes

and every turn after it failed. Measured live on a 14-turn variation matrix:
turns 11-14 all returned nothing once the prompt crossed ~55k.

What the flag protects is vision spans -- a bidirectional mask over an image
would break if the span were split across forwards. (The MLX language model
implements no such mask today, _make_masks is causal-only, so a split is
currently output-identical; that is a fact about today, not a licence.)

_media_chunk_boundaries already guarantees no boundary lands inside a media
run: a run starting after the cut moves whole to the next chunk, and a run
already open at the cut extends the chunk to cover it -- even when the run is
longer than the nominal chunk size. So the intent is satisfiable without
refusing to chunk.

The invariant is now VERIFIED rather than assumed: before forwarding, every
boundary is checked against every run, and if any boundary would split a span
the request falls back to the one-shot path with a log line naming the run.

Tests: the old test asserted the kill-switch behaviour and now asserts the
opposite, with the metal::malloc failure in its docstring so the reason
survives. Two new tests pin the invariant directly, including a media run
LONGER than the chunk size.
J
Jinho Jang committed
0abdd34cd3caef76c629c557ca36cb52eef2b9ae
Parent: d1c7880