SIGN IN SIGN UP

fix(diffusion): apply compute_dtype via autocast when FSDP2 skips (#3589)

* fix(diffusion): apply compute_dtype via autocast when FSDP2 skips sharding

Split model.torch_dtype/model.compute_dtype relies on FSDP2's
MixedPrecisionPolicy to cast parameters to the compute dtype. On a
single-rank world or mesh, FSDP2Manager.parallelize skips sharding, so
that cast never happens: parameters stay in torch_dtype while the
flow-matching step still casts its inputs to compute_dtype. The forward
pass then mixes dtypes and fails with

  RuntimeError: Input type (c10::BFloat16) and bias type (float) should
  be the same

on the transformer's fp32 patch embedding. The same config works on 2+
GPUs, and five shipped diffusion example configs use this split
(torch_dtype: float32 with compute_dtype: bfloat16).

Wrap the forward pass in torch.autocast on that path, following the
autocast approach the retrieval and dLLM recipes already use for
FSDP2Config.autocast_dtype. Resident parameters and their gradients stay
in torch_dtype, so optimizer state is unchanged. Numerics are close to
but not identical to the sharded path, since autocast keeps reductions
such as normalization in fp32 rather than casting every parameter.

Extract the skip condition into fsdp2_sharding_enabled() so the recipe
checks the same predicate FSDP2Manager acts on instead of duplicating it.

LLM/VLM recipes drop mp_policy at world size 1 the same way, but degrade
silently to fp32 compute rather than failing; that is left for a separate
change.

Fixes #3578

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>

* test(diffusion): stub _autocast_dtype in the metrics recipe test

test_run_train_validation_loop_uses_hot_path_and_logs_perf_metrics builds the
recipe with object.__new__ and assigns each attribute the training loop reads,
bypassing setup(). The loop now also enters _autocast_context(), which reads
self._autocast_dtype, so the test failed with AttributeError on both
L0_Unit_Tests_CPU and L0_Unit_Tests_GPU_1_of_2.

Set it to None alongside the existing transformer_engine_fp8=False stub, which
is the direct analogue: both disable a per-forward wrapping context. The recipe
class intentionally has no class-level attribute defaults -- setup() owns every
attribute -- so the stub belongs in the test rather than in the recipe.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>

---------

Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Z
Zeyu Zhou committed
3efbebe6f60f8ba441b499c951f83090d753736c
Parent: bd58fd6
Committed by GitHub <noreply@github.com> on 8/20/2026, 8:20:33 PM