Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.
docs: expand DDP metric synchronization guidance (#21685)
Restructure the "Synchronize validation and test logging" section in accelerator_prepare.rst into a problem-framing intro plus three subsections (sync_dist, TorchMetrics, manual all_gather), a decision table, and a common-pitfalls list. Directly addresses the custom-metric case: accumulate per-step outputs, call all_gather at epoch end, and compute the metric. The "my compute runs N times" confusion is called out and resolved — after all_gather every rank holds the same data, so the redundant compute is cheap and correct; only self.log needs the rank_zero_only guard. Refs #20117 Co-authored-by: Deependu <deependujha21@gmail.com>
C
c-pozzi committed
932b7e3e76ea96481003ceacb13c3e8194f281d2
Parent: a8d32b0
Committed by GitHub <noreply@github.com>
on 5/27/2026, 5:23:44 PM