fix(mllm): keep every image in the message that carried it
THIS is why multimodal prefix reuse was permanently 0%. It was never a cache
defect.
mlx_vlm's whole-list builder finds the LAST user message and re-attaches EVERY
image to it, stripping them out of the messages that actually carried them:
extract_text_from_content drops the image part, then
`skip_image_token=not is_target` re-emits all num_images placeholders on
last_user_idx. For a single-turn request that is a no-op -- which is exactly
why it looked correct, and why every existing probe missed it: they all used
ONE fixed message list, the shape where the reposition is invisible by
construction.
Extend the conversation by one turn and the image MOVES. Measured live on
Qwen3.8-27B VL through /v1/chat/completions:
request A (4 msgs, image in the last user msg):
n=2456 media_tokens=72 first_media_at=84
request B (A + assistant + user, image STILL in message 3):
n=2478 media_tokens=72 first_media_at=2396
Same image, same 72 expanded media tokens, 2312 tokens apart. Every token
after the image shifts, so no block hash can match and the common prefix
collapses to the pre-image header (~84 of 2478 tokens, about 3%). That is the
whole of the "media suppresses reuse for the entire conversation" behaviour,
and it mechanically nullified the media prefix-cache allow-list for qwen3_5*
and gemma4* -- two of its three families.
It also misattributes images on a FIRST request: two images in two different
messages both get bunched onto the last user message.
The images-only branch now builds per message, reusing mlx_vlm's own
per-family get_message_json so every family's literal in-text image syntax
(<start_of_image>, <image>, <|image_N|> ...) is produced by exactly the same
code as before. The only thing that changes is WHICH message carries the
placeholders. Images supplied through the separate images= kwarg with no
owning message part keep the old last-user-message convention, since there is
nothing better to key them on.
Tests assert the CROSS-REQUEST invariant, which is the only one that can see
this: the image's message index must be identical for a conversation and for
that same conversation plus one turn. One test also pins that upstream
mlx_vlm still has the defect, so if it is ever fixed there this local builder
can be reconsidered instead of lingering unexplained.
Suite: 8999 passed, 2 failed -> both addressed here (one was a stub in
test_mimo_v2_media_runtime that predated _media_forward; the other is the
bundled-python drift gate, expected until the re-bundle). J
Jinho Jang committed
b0419174c9bd31e1ce6ec930125e7d4cd15e2151
Parent: 1dfdd3b