SIGN IN SIGN UP

[Bugfix][Router] Sanitize prefiller payload in disagg-prefill (stream, min_tokens) (#971)

In disaggregated-prefill mode, send_request_to_prefiller forces
max_tokens=1 but otherwise inherits the client's request body. Two
fields can break the prefill step:

1) stream

When the client sends a streaming request (stream=true), the prefiller
returns text/event-stream, which aiohttp's response.json() refuses to
decode by default, raising:

    aiohttp.client_exceptions.ContentTypeError: 200, message='Attempt to
    decode JSON with unexpected mimetype: text/event-stream; charset=utf-8'

As a result, every streaming request fails the prefill step. This is
trivially reproducible with 'vllm bench serve' (which sends stream=true
by default).

The orchestrated path (route_orchestrated_disaggregated_request) already
handles this correctly by forcing stream=False on the prefill request.
This change ports the same fix to send_request_to_prefiller so both code
paths behave consistently. Streaming is pointless here anyway since
max_tokens=1, and the prefill response body is not consumed by the
caller in the disagg-prefill flow.

2) min_tokens

vLLM's SamplingParams rejects requests with min_tokens > max_tokens. If
the client sends min_tokens > 1, the prefiller would return 400 since
we just forced max_tokens=1. Drop min_tokens from the prefill payload
to keep the prefill step robust against arbitrary client inputs.

Together these two changes make the prefiller payload safe regardless
of what the client sent, while leaving the original request_json used
by the decode phase untouched (req_data is a shallow copy).

Signed-off-by: liucunzhuang <80206067+lcz1998@users.noreply.github.com>
Co-authored-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com>
L
lcz^_^ committed
010d4888d619acb00541d10eddd78a66eac3c272
Parent: b3e3cb8
Committed by GitHub <noreply@github.com> on 6/16/2026, 9:01:16 PM