Resolve continuous batching config against the text config for composite models (#48299)
PagedAttentionCache and the CB config resolution read decoder attributes (num_key_value_heads, num_hidden_layers, head_dim, ...) from the top-level config, which does not carry them for composite configs (vision-language models keep them on text_config). Continuous batching on any ForConditionalGeneration model crashed with 'num_key_value_heads or num_attention_heads could not be found in the config'. Resolve against config.get_text_config(), a no-op for text-only configs.
Q
Quentin Gallouédec committed
598d8ba8baaec7fec5a22da0e2844c7bf4ea20e1
Parent: eb4d9e2
Committed by GitHub <noreply@github.com>
on 8/26/2026, 2:08:50 PM