fix(fsdp): uniform reduce dtype and EP-local expert gradients (#3540)
* fix(fsdp): keep reduce-scatter gradient dtype uniform under accumulation
FSDPParamGroup.post_backward feeds foreach_reduce from two slots that hold
different dtypes: unsharded_accumulated_grad, upcast to reduce_dtype (e.g.
fp32) for parameters that received a gradient during an earlier no-sync
micro-batch, and the raw unsharded_param.grad in param_dtype (e.g. bf16) for
parameters whose gradient joined only now. A group holding both trips
AssertionError: FSDP reduce-scatter expects uniform gradient dtype
but got {torch.bfloat16, torch.float32}
Context parallelism makes this reachable: patch_fsdp_unused_param_reduction
fills locally unused parameters with a param-dtype zero, which lands next to
peers that already hold reduce-dtype accumulations. Any cp_size > 1 recipe
with gradient accumulation can reach that state (NVBug 6599894: GLM-5.2
glm_moe_dsa cp8 plus pp4, which never completes step 0).
Widen the minority gradients inside foreach_reduce itself, the last point
before the reduce. At that point the gradients are already plain local
tensors, FSDP2's own bookkeeping is untouched, and foreach_reduce copies them
into a reduce_dtype buffer regardless, so no value changes. Uniform groups
pass straight through, so the upstream assertion still catches genuinely
inconsistent gradients such as fp8 weights that fail to produce
higher-precision ones.
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
* fix(fsdp): reduce unused expert gradients from local tensors
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
* fix(fsdp): localize residual DTensor gradients
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
* fix(fsdp): prefer public unused-param reduction
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
---------
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> A
Alexandros Koumparoulis committed
79fd3a4190fc450bd6d2ecdc20c8398fb725e458
Parent: 0d3c798
Committed by GitHub <noreply@github.com>
on 8/15/2026, 12:26:23 AM