SIGN IN SIGN UP

Add an opt-in per-frame pixel cap (cap_pixels_per_frame) to the Qwen3-VL video processor (#48071)

* Add max_pixels_per_frame to Qwen3-VL video processor

The video pixel budget (size.longest_edge) is spent across all sampled
frames, so clips that sample fewer than max_frames frames keep
near-native per-frame resolution and a short clip can cost almost as
many tokens as a long video. The new optional kwarg caps the effective
budget at num_frames * max_pixels_per_frame, making token cost scale
with clip duration when set. Default None preserves current behavior.

* Floor the frame count at 32 when applying max_pixels_per_frame

Serving stacks probe processors with tiny synthetic clips (vLLM's boot
memory profiling uses 2-frame dummies); capping those at
num_frames * max_pixels_per_frame collapses them to near-minimum
resolution. Apply the cap against max(num_frames, 32) so short and
synthetic clips keep a usable floor while longer videos stay
duration-proportional.

* Regenerate CohereCompass video processor from modular source

CohereCompassVideoProcessor inherits from Qwen3VLVideoProcessor, so
the modular conversion check picks up the new max_pixels_per_frame
kwarg. Commit the regenerated file so it matches what
check_modular_conversion.py produces.

* Replace max_pixels_per_frame with a cap_pixels_per_frame boolean

Per review: instead of a numeric knob, a boolean that applies the
qwen-vl-utils per-frame cost formula (768-patch frame ceiling or the
budget's even share per frame, floored just above min_pixels). Unset
warns that the capped behavior becomes the default in v5.22, when the
argument will be removed; passing False keeps today's behavior silently.

* Apply ruff formatting

* Spell Qwen3VL in the deprecation warning so modular conversion renames it

* Add cap_pixels_per_frame to the Qwen2VL video processor

* Regenerate VideoLlama3 video processor from modular source

* Address review: parameterize caps as class attrs, inherit Qwen2VL _preprocess, drop stray attrs

- max_video_tokens (768) and video_total_seq_len (128000) become saved class
  attributes instead of magic numbers
- Qwen3VLVideoProcessor now inherits Qwen2VLVideoProcessor and overrides only
  resize and sample_frames; _preprocess (with the deprecation warning) is
  copied by modular conversion
- VideoLlama3 no longer inherits the cap attributes or kwargs; its resize
  already spreads the budget across frames
- document min_frames/max_frames in Qwen3VL kwargs (silences the
  auto_docstring runtime lint)

* Override get_num_of_video_patches for Qwen3VL's temporal smart_resize

The copy inherited from Qwen2VL called this module's t*h*w smart_resize with
per-frame arguments. Override with the temporal signature and apply the
per-frame cap so the patch count matches what resize actually produces.

* small unification in naming

---------

Co-authored-by: raushan <raushan@huggingface.co>
D
Devon Krisman committed
27e7f6c81a67bae6f706098083cbe9c64aba8def
Parent: 53e3152
Committed by GitHub <noreply@github.com> on 8/26/2026, 8:01:54 AM