ci: add gradient-determinism rule to PR review guidelines (#3392)
The automated reviewer missed a P1 on #3374: replacing a per-group
split/cat loop in `_apply_bias` with `torch.repeat_interleave` preserved
the forward result but turned the backward into an unordered atomic
accumulation, making the trainable expert-bias gradient nondeterministic
run-to-run on CUDA.
The existing "Distributed autograd and gradient handling" section did not
fire, for two reasons. It is gated on training-loop constructs
(`backward()`, `no_sync`, FSDP, hooks, scaler, optimizer, DTensor,
collectives), none of which that diff touches. And every check under it
asks whether the gradient is wrong, whereas this gradient stays
mathematically correct and merely stops being reproducible, so
tolerance-based reference comparison cannot see it.
This adds one ungated top-level rule that triggers on the shape of the
diff rather than the subsystem. The op classification was verified on an
RTX 5880 Ada (torch 2.10, bf16, fixed upstream gradient, bitwise compare
over 12 runs):
nondeterministic backward: repeat_interleave with tensor repeats,
index_select and gather when the indices repeat
deterministic, must not be flagged: expand and implicit broadcast,
split+cat loops, embedding, advanced indexing x[idx]
nondeterministic in the forward only: index_add_, scatter_add_,
index_put_(accumulate=True), and bincount, which has no gradient
Nothing in that set raises under torch.use_deterministic_algorithms(True);
it silently selects a slower kernel, so determinism-mode CI does not catch
this class either.
Replayed on #3374 at f8095c30, the current prompt mentions determinism in
0/2 runs; with this rule 2/2 report the finding with the correct mechanism
and the sum().backward() test gap. Replayed on #3387, which fixes the bug
with a custom autograd Function, the rule correctly stays silent.
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> A
Alexandros Koumparoulis committed
e81ea89bd409b8e61412fa2bff9f33b4dbf980de
Parent: ee0b226
Committed by GitHub <noreply@github.com>
on 8/4/2026, 2:14:03 PM