SIGN IN SIGN UP

fix(glm): align and diagnose cross-framework router parity (#3635)

* fix(glm): diagnose cross-framework router drift

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* test(glm): calibrate routed parity profiles

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* test(ci): add parity shape and router diagnostics

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* fix(ci): auto-report shortened parity gate shape

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* docs(ci): explain GLM parity gate rationale

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* docs(ci): clarify parity diagnostic purposes

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* test(ci): enable GLM-4.5-Air robustness coverage

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* docs(ci): clarify router diagnostic support

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* fix(ci): size GLM-4.5 robustness batches for EP32

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* test: compute parity cosine in float64

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* test: keep GLM-4.5 robustness experiment-only

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* fix(glm): own router precision and parity diagnostics

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* test(ci): enable GLM-5.2 checkpoint robustness, AutoModel-side phases

Enable the checkpoint-robustness suite for GLM-5.2 with the cross-framework
phases (vanilla-HF source parity and HF reload) skipped: they load the 1.5TB
checkpoint in a single rank-0 process (~50 min just to load), idling the
32-node allocation. Cross-framework parity was verified out-of-band via a
single-node CPU-offload vanilla-HF reference against the EP64 AutoModel stack
on the bundled parity document: mean KL 0.0049 at 512 tokens (passes the
standard profile); 0.0071/0.0065 at 2K/4K with p95 within standard and no
divergent-token tail (median ~0.002, max 0.57) - a 78-layer bf16 noise floor,
not routing instability.

Also set model_save_format/save_consolidated in the checkpoint block: the
train_and_save phase runs the recipe's checkpoint config verbatim, and without
save_consolidated: final the save produces sharded safetensors plus an offline
consolidate.sh only, leaving the reload phase an empty consolidated directory
(verified empirically; inline distributed consolidation of the 1.4TB export
measured 4m34s end-to-end on 8 nodes, well within the CI time budget).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* test(glm): update backend assertions for fp32 routing

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* fix(glm): harden parity diagnostics

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* fix(checkpoint): harden GLM resume diagnostics

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* fix(checkpoint): stabilize GLM5.2 resume validation

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* refactor(deepseek): require explicit MLA latent norm epsilon

The shared MLA is reused by DeepSeek-V3, Mistral 4, and GLM MoE Lite, so a
hidden epsilon default would let a future family silently inherit numerics it
never chose. Each Block now states its checkpoint contract explicitly; all
current upstream implementations construct the two latent RMSNorms with the
Hugging Face reference default of 1e-6 rather than config.rms_norm_eps, and
the HF config carries no field for it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

---------

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Y
Yuhe Zhang committed
37ff7feacbfdc14a1c69d178eb3f9a9119f1ab0d
Parent: 792a6cd
Committed by GitHub <noreply@github.com> on 8/28/2026, 6:15:15 PM