SIGN IN SIGN UP

Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.

0 0 104 Python

docs: expand DDP metric synchronization guidance (#21685)

Restructure the "Synchronize validation and test logging" section in
accelerator_prepare.rst into a problem-framing intro plus three
subsections (sync_dist, TorchMetrics, manual all_gather), a decision
table, and a common-pitfalls list.

Directly addresses the custom-metric case: accumulate per-step outputs,
call all_gather at epoch end, and compute the metric. The "my compute
runs N times" confusion is called out and resolved — after all_gather
every rank holds the same data, so the redundant compute is cheap and
correct; only self.log needs the rank_zero_only guard.

Refs #20117

Co-authored-by: Deependu <deependujha21@gmail.com>
C
c-pozzi committed
932b7e3e76ea96481003ceacb13c3e8194f281d2
Parent: a8d32b0
Committed by GitHub <noreply@github.com> on 5/27/2026, 5:23:44 PM