fix: EP collective deadlock with variable-length token counts (LoRA flavor) (#3631)
* fix: EP collective deadlock with variable-length token counts (LoRA flavor) GroupedExpertsLoRA.forward still gathers tokens across the EP mesh with DTensor.from_local(x, [Shard(0)]).full_tensor(), which assumes uniform token counts across ranks. PR #1365 (8f2b685) fixed this in GroupedExperts but not in the LoRA subclass, so LoRA on MoE experts with unpacked or unpadded batches deadlocks in NCCL on the first MoE layer. Port of #1365's fix to lora_experts.py: pad+all_gather+trim on the way in, all_reduce+narrow on the way out, gradient anchor so every rank enters the backward collectives. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Aarni Koskela <akx@iki.fi> * test(moe): cover ragged LoRA expert parallelism Signed-off-by: HuiyingLi <willwin.lee@gmail.com> --------- Signed-off-by: Aarni Koskela <akx@iki.fi> Signed-off-by: HuiyingLi <willwin.lee@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: HuiyingLi <willwin.lee@gmail.com>
A
Aarni Koskela committed
a12b237ed1a744b99a653f09b18ab9f5c4eb3a10
Parent: e2a3ee6
Committed by GitHub <noreply@github.com>
on 8/23/2026, 10:51:35 PM