Add MuseGlimmer (#47867)

* add-new-model-like gemma2

* Copy trust_remote_code implementation

* Start modular for LLM

* draft image model and config

* prob packed attn will be better in terms of perf, than padding everything

* come back on monday, keys match but logits dont

* Fix guard order for loading

* Fix rms norm eps

* some config keys got lost/renamed?

* fix permute for rope

* images weren't being passed at all, fix it

* oops, wrong dim

* some defaults are not correct

* temporal patch comes first?

* still not identical but should be right to match align-corner=False

* so positions are flipped, and we need to pad in zeros

* and here comes video processing

* correctness updates

* original expectations + tester (with local paths for now)

* small changes

* more cleanup + some renaming

* enable_circleci

* enable_circleci

* check_install

* fix circleci: use large resource class and clean up parallelism expression

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* halve pytest num workers to fit large resource class

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* check_install

* Conversion script

* Remove on-load renames

* Fix auto mapping

* skip some tests for now

* Update IMAGE_PROCESSOR_MAPPING_NAMES

* more modular for vision

* config docs

* Make lists explicit

* Remove legacy key support

* Remove no longer used keys

* Use one of the supported layer_types

We only check for "full_attention"

* Remove legacy flat keys

* Apply modular

* Adapt converter

* Fix typo after rename

* make style

* more renames to be like qwen

* i will better fix on main, not needed anymore here

* tests - rename config keys

* can delete no? we got a multimodal class working now

* why does this not crash more often?

* make things green up a bit

* check not running

* check will run with [run-circleci] commit message

* check not running

* modular video processing

* biiig address of comments and review

* Fix Onyx chat generation prompt

* style

* [run-circleci]

* config fixes and docstrings

* [run-circleci]

* clean up + unbloat with autodoc

* format

* [run-circleci]

* new naming (no need to reconvert)

* Remaining downsample_factor reference

* onyx(modular): port Meta tool-calling template + response_schema + eos into modular structure

Ports the model-team-owned pieces onto HF's modular onyx structure (onyx-modular):

- onyx_chat_template.py: the authoritative ATEM tool-calling chat template
  (byte-identical to internal genai/msl/guac/hf). Wired into processing_onyx.py
  via build_chat_template(); replaces the modular stub template.
- Generation prompt: OVERRIDES the modular hardcode '<|start|>assistant to=user<|message|>'
  with the bare '<|start|>assistant'. The to=user hardcode structurally prevents tool
  calls (calls emit on to=<tool> channels). Documented inline; decision to be shared with HF.
- convert_onyx_weights_to_hf.py: bake ONYX_RESPONSE_SCHEMA onto the tokenizer
  (parse_response output parser, byte-identical to internal) + add
  write_generation_config() emitting generation_config.json with
  eos=[200001,200008] (drops <|eom|>=200007, the proven eos bug).

Verified: template renders bare gen-prompt + ATEM tool-defs + reasoning-strength +
valid-recipients + multimodal image sentinel; response_schema md5 == internal; all files compile.

NOT yet done (follow-ups): safety-probe gate port (model.model.layers -> modular
language_model path), make fix-copies/style, end-to-end convert+generate test.

* onyx(modular): port safety-probe gate for the TARGET (modular) architecture

Adapts Sid's --safety_probe gate (D113317217) to HF's modular Onyx structure, where the
deliverable arch is OnyxForConditionalGeneration (image-text-to-text), not OnyxForCausalLM.

The probe concatenates per-layer decoder hidden states, so it must reach the text decoder
layers. In modular they nest as:
    OnyxForConditionalGeneration.model (OnyxModel) -> .language_model (OnyxTextModel) -> .layers
whereas the old text-only export was model.model.layers. Hardcoding model.model.layers
(as landed) breaks on the modular arch.

Changes (deliverable_hf/inference_hf.py, staged copy of genai/msl/guac/hf/inference_hf.py):
- add _get_decoder_layers(model): resolves the decoder ModuleList across arches
  (model.model.language_model.layers -> model.model.layers -> model.language_model.layers),
  preserving layer order 0..N-1 which the probe requires; raises clearly if none match.
- probe uses _get_decoder_layers(model) instead of model.model.layers.
- loader: try AutoModelForImageTextToText (modular target, MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING)
  then fall back to AutoModelForCausalLM, so the script works against either deliverable.

Verified: _get_decoder_layers resolves 52 layers for both modular and causal mocks and
raises on a bad model; file compiles. NOT yet run end-to-end against a real converted
modular checkpoint + the rc1__v07e probe .pt (follow-up).

* fixing max token lenght

* reverting formatting changes

* Increase model max length to 131072

* onyx(modular): fix max_position_embeddings 16384 -> 131072 (RoPE cache crash on convert)

build_config() and the OnyxTextConfig default set max_position_embeddings=16384, but the
tokenizer model_max_length and generation max_length are 131072. max_position_embeddings
sizes the RoPE cos/sin cache (max_seq_len_cached), so any sequence >16384 crashes on RoPE
position indexing. A per-checkpoint config.json edit (16384->131072) was applied manually,
but every fresh conversion re-emitted 16384 and reintroduced the crash.

This is the real fix: emit 131072 from build_config() + make it the OnyxTextConfig default,
consistent with model_max_length/max_length=131072. Vision config (32*32) unchanged.

Verified: build_config().text_config.max_position_embeddings == 131072; OnyxTextConfig() default == 131072.

* vLLM transformers modeling backend changes

* rename

* parallel tool call line - remove

* onyx: move chat template to standalone chat_template.jinja file

Extract the Jinja chat template out of the inline ONYX_CHAT_TEMPLATE Python
string constant into a sibling chat_template.jinja file, loaded at import time.
Matches the HF convention (cf. the Gemma release, which reads chat_template.jinja
via cached_file rather than embedding it in the converter). Addresses PR review
feedback to move the template off the .py constant before release.

- onyx_chat_template.py now reads chat_template.jinja (byte-identical output;
  build_chat_template() and ONYX_MM_CHAT_TEMPLATE unchanged for consumers)
- setup.py: add **/*.jinja to package_data so the template ships in wheels
- template content is Ruan's 662a1072d5 (parallel-tool-call line removed)

Verified: load + greedy generation of the converted rl_v1 checkpoint on GPU
renders correctly via the file-loaded template (plain chat stops on eos;
single tool call emits a well-formed <atem:invoke> and stops on eos).

* onyx(modular): sync max_position_embeddings in modular source + regen

Commit cb5a4d03a7 bumped max_position_embeddings 16384->131072 in the generated
configuration_onyx.py but not in the modular source (modular_onyx.py), leaving
check_modular_conversion failing. Fix the value in OnyxTextConfig in the modular
source and regenerate, which also picks up a stale docstring ordering in
modeling_onyx.py (image_grid_thw/video_grid_thw moved after labels, matching
current converter output).

check_modular_conversion now passes (RC=0); OnyxTextConfig().max_position_embeddings
== 131072.

* onyx: load chat template from the model Hub repo at convert time

Follow the HF convention (cf. the Gemma release, which reads chat_template.jinja
from its model repo via cached_file) so the model repo is the single source of
truth for the template. The converter now fetches chat_template.jinja from
ONYX_HUB_REPO and bakes it into the converted checkpoint, falling back to the
copy packaged with transformers when the Hub is unreachable (offline / CI).

ONYX_HUB_REPO points at the temporary early-release repo (someorgtoo/onyx_early)
and must be updated to the final public repo before release.

Runtime processing_onyx.py is unchanged: it still reads the packaged
chat_template.jinja locally (no network at import). Only the convert script
touches the Hub.

* onyx: migrate response parser to declarative response_template

Replace the legacy x-regex ONYX_RESPONSE_SCHEMA with the new declarative
response_template format (transformers 5.x), addressing PR review feedback that
the x-regex response_schema is being deprecated in favor of the streaming-capable
response-parsing API.

Fields map 1:1 to the Onyx channels:
- reasoning_content: open_pattern to=self<|message|>, close <|eom|>, text
- tool_calls: open_pattern captures the <atem:invoke> name, xml-inline parses
  each <atem:parameter> (value as lax JSON), repeats -> list (parallel calls),
  transform wraps into {type: function, function: {name, arguments}}
- content: open_pattern to=user<|message|>, close on <|eot|>/<|eom|>, text

Channel-scoping is now structural instead of regex-substitution based: while an
explicit region is open the parser only watches that region's close, so an
<atem:invoke> echoed inside reasoning or the final answer is body text, not a
spurious tool call. Assigned as tokenizer.response_template (was response_schema).

The old schema joined multiple to=self blocks; the model emits at most one
to=self block per turn (verified across 1768 captured turns), so a single-string
reasoning_content region is behaviorally equivalent and matches the standard
reasoning_content contract.

Verified end-to-end: converter template validates; tokenizer_config.json gets
response_template (no legacy response_schema); reload + tokenizer.parse_response()
on a real model generation yields correct reasoning_content + tool_calls; parallel
calls, nested-object args, mixed scalars, and echoed-in-reasoning all parse
correctly. check_modular_conversion RC=0.

* Image placeholder -> <|patch|>

* downsample_factor -> merge_size

* onyx: load chat template from hf/ subfolder of onyx_early_v2

* onyx: remove in-tree chat template, load only from model repo

* draft, not checked yet, needs weight conversion

* maybe tmrw

* make replace image token match reference format

* commit before i mess it up again. Shapes match but cropping should go differently

* greedy decoding matches assistant, cand suggestion are garbage

* needs smth to actually compare logits, i ma just blind shooting based on code inspection...

* [run-circleci]

* permute rope and fix some cropping

* oops

* interleave

* this gets hugher acceptance, and i need to have lunch

* add proper conversion script

* nit

* unmerged emebds

* revert back interpolation and image processor

* revert unrelated

* do same as in main

* just copy test from glm for now, quick sanity check

* revert unrelated

* add the Text prefix everywhere

* small style

* natural module order in files

* remove unused rope function

* [run-circleci] style

* [run-circleci] fix record output

* fix audio features

* [run-circleci] empty commit for CI

* simplify and fix cache past recording

* un-merge the norm for dflash

* rename

* put it after mtp in the file

* crop correctly in dflash

* oupsi forgot to remove

* Update src/transformers/models/onyx/processing_onyx.py

Co-authored-by: Aritra Roy Gosthipaty <aritra.born2fly@gmail.com>

* Update src/transformers/models/onyx/processing_onyx.py

Co-authored-by: Aritra Roy Gosthipaty <aritra.born2fly@gmail.com>

* revert and fix cache and position ids, use target states for cache

* simplify attention

* frogot to revert , comment doesn't match code

* fix inference and add docstring

* oupsi double equal

* correct mask similar to diffusion gemma

* clean up

* revert all that is left

* tests, yet not integration tests

* fix integration test

* permute rope

* update prefix image

* doc and remove unused init args

* update arg name

* a bit more doc and variable name matching

* more doc in the model for sensitive aspects

* fix bug on q_offset, and use native mask API

* simplify a bit

* style

* no checks at all if speculation_type is dflash

* rename

* update test

* rework and finish tests

* namings/configs

* minor fixes

* Rename onyx_assistant

* [revert] get back image delimiters

* style

* update test

* Final location of chat template

* align the config defaut value

* no need to just call the parent it already exists

* finalize the renaming

* list cannot be default

* allow sampling in dflash

* add doc and public inits

* update repo name

* assistant checkpoint name as well

* skip

---------

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: raushan <raushan@huggingface.co>
Co-authored-by: Pablo Montalvo <pablo.montalvo.leroux@gmail.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Raushan Turganbay <raushan.turganbay@alumni.nu.edu.kz>
Co-authored-by: Benjamin Burtenshaw <benjamin_burtenshaw@users.noreply.github.com>
Co-authored-by: Beto de Paola <betodepaola@meta.com>
Co-authored-by: albertodepaola <albertodepaola@gmail.com>
Co-authored-by: ruanrms <ruanslv@gmail.com>
Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com>
Co-authored-by: Aritra Roy Gosthipaty <aritra.born2fly@gmail.com>