SIGN IN SIGN UP

Resolve continuous batching config against the text config for composite models (#48299)

PagedAttentionCache and the CB config resolution read decoder attributes (num_key_value_heads,
num_hidden_layers, head_dim, ...) from the top-level config, which does not carry them for
composite configs (vision-language models keep them on text_config). Continuous batching on any
ForConditionalGeneration model crashed with 'num_key_value_heads or num_attention_heads could
not be found in the config'. Resolve against config.get_text_config(), a no-op for text-only
configs.
Q
Quentin Gallouédec committed
598d8ba8baaec7fec5a22da0e2844c7bf4ea20e1
Parent: eb4d9e2
Committed by GitHub <noreply@github.com> on 8/26/2026, 2:08:50 PM