[Glm 5.3 Flash] GLM 5.3 Flash Support (#48342)

* [Model] Add GLM5-Next model support

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Apply new model config

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Add fp8 loading

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Fix error to pass tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Add default mlp_layers_types

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Refactor codes by inheriting

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Update from new config.json

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Add new Indexer support

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Remove unnecessary changes

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Update Kpool related

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* feat: Add Kpool

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* fix: Add skip rope in indexer

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* start refactoring

* more cleanup, mainly lin attn

* p1

* more fla fixes

* more refactoring

* refactor conv1d

* oops

* last things for today

* start refactoring indexer

* modularize the attention init for the most part and properly refactor rope/nope

* simplify attention a bit

* lets wrap it up

* integrate kernels, needs some other PR merges

* tmp kernels

* apply patch and add TODOs

* fix hooks

* how did they get missed?

* part1

* nits

* last fixups

* fix

* more simplify

* fixups

* rename

* update kernels to official ones

* PoC rewrite of the indexer, matches non padding generation / forward - but too much AI need to properly rewrite / comment on top of that

* modular and add the fp8 fix locally for now

* slightly refactored indexer

* finished indexer for now

* fixup cache init + add ep plan

* lazy set device

* revert this

* point to staging for now

* sync with main

* strong modular inheritance

* start config cleanup

* Update src/transformers/models/glm5_next_vl/configuration_glm5_next_vl.py

* correct modular, we need to potentially fix get placeholder mask

* nit

* tests for text only 70B

* small nits

* check repo fixes (text only tbh

* modular config

* fix repo

* modular vl config and default values

* fix repo

* current state with tests

* quick skips, 2 tests remaining

* last fixes

* check repo fixes

* missed

* little fixes

* bump version of liger (fixes device context)

* step 1 vlm only

* config

* rename

* more fixups

* make fix and check

* fix

* Add indexer and kpool

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* fixup tests to new state

* fix repo

* clean

* fix

* fix

* make yarn scale more modular friendly and small fixups

* fixup

* Add Glm5NextVideoProcessor

Glm5NextVideoProcessor(GlmgaVideoProcessor) overrides sample_frames with the GLM-5-Next frame-sampling algorithm and inherits glmga _preprocess unchanged. Wire glm5_next -> Glm5NextVideoProcessor in the video auto-map and export it; image processor stays GlmgaImageProcessor.

* modular version and fixups

* tests and repo fixes

* fix repo

* nits and remove bool call from recurrent mask for now

* fix wrong type hint

* update config

Co-authored-by: Vasqu <antonprogamer@gmail.com>
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Apply SwiGLU clamp in GLM-5-Next vision encoder

Port the GLM-5-Next VIT changes from sglang (b385c5ad / 66e19224 /
1059d34c) to the transformers side.

- Add Glm5NextVisionConfig (model_type glm5_next_vision) carrying
  swiglu_limit, resolved from vision_config or the text_config fallback.
- Add clamped Glm5NextVisionMLP / Glm5NextVisionPatchMerger and the
  Glm5NextVisionBlock / Glm5NextVisionModel that use them, all via
  inheritance from the shared glm_ocr vision classes.
- Register glm5_next_vision so AutoModel/AutoConfig resolve to the
  clamping vision tower.

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* Update GLM5 Next vision config and tests

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* pass test

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* fixes for main compatibility

* new update fixes

* post separate vision model fixes

* fix recurrent gdn fallback for fla

* fix repo

* fix check repo

* feat(glm): update image and video processing

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* update

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* update for new cfg

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* update from inner

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* update from review

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* refactor

* fix

* docs: rename GLM-5-Next to GLM-5.3-Flash

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* img tests

* vid tests

* fix repo

* add comment

* update

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>

* last fixes

* revert this, moved to separate PR

* fix rules

* fix

* fix toctree

* docs: update glm5-3-flash

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
Co-authored-by: zRzRzRzRzRzRzR <Yuxuan.Zhang2@liverpool.ac.uk>
Co-authored-by: Ilyas Moutawwakil <57442720+IlyasMoutawwakil@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* fixups

---------

Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
Co-authored-by: Vasqu <antonprogamer@gmail.com>
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
Co-authored-by: zRzRzRzRzRzRzR <Yuxuan.Zhang2@liverpool.ac.uk>
Co-authored-by: Ilyas Moutawwakil <57442720+IlyasMoutawwakil@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>