Add Qwen4Exp model (#48337)

* support Qwen4-Exp.

* Fix placement of concatenated checkpoint tensors

* Align Qwen4-Exp checkpoint conversion

* Simplify Qwen4-Exp model integration

* Add Qwen4-Exp parallel coverage and documentation

* revert some equivalent operation

* update qwen4_exp doc.

* fix

* no need to change qwen3.5

* sets activation directly inside norm

* remove specific head models

* native rotary

* simplify

* remove overwrite

* remove non compile criteria

* move outside the class

* remove constants

* simplify config

* indexer is always present

* indexer needs to train

* fix and simplify the selection of valid indices in indexer

* remove external functions

* simplify indexer further

* replace einsum with matmul

* start vectorizing

* simplify

* always use a mask for simplicity

* create and combine mask much more easily in indexer

* simplify hyper connections

* fix all prefixes

* fix ple input by reversing embeddings instead

* capture hidden states

* follow signature order

* always forward both masks

* fix and simplify past context in ngram embedding

* fix

* make do_natural_key more readable

* combine norm and grouped norm

* fix shapes

* fix qsa check

* fix mask

* style

* add a few decorators

* update minimal test

* adapt to newest config

* remove unnecessary test

* combine related test

* fix index mask broadcast error

* fix gdn previous state check when ple is enable

* avoid update conv state in place

* confirmed hybrid_indexed was not used

* fix rope test

* use vlm tester to pass ci

* rewrite layer type in config

* no need to reslice

* short conv up to latest standards

* dilution not supported in kernels 😭

* remove unused attr

* vectorize qsa indexer!!!

* qsa can use a separate rotary for simply

* add qwen_sparse_attention to support block key store!!!

* update test

* modular based on latest main

* make quality happy

* keep simple for now, generalize all library indexers later

* do not keep cos/sin in the cache for memory

* a bit cleaner

* extend 2d position_ids

* ignore rule

* simplify conversion Ops

* fix loading and tests

* do not use attn in tp_plan cause num_kv_head=2

* remove created file

* oupsi

* fix chunked prefill

* fix positions

* fix reverse loading

* improve conversion

* slightly clearer key ordering

* shard the embedding on dim1

* revert test

* force conversion on cpu for device_map

* make things work on simple hardware

* small fix

* oupsi

* latest upstream modular

---------

Co-authored-by: 弥道 <roy.zb@alibaba-inc.com>
Co-authored-by: JJJYmmm <1650675829@qq.com>