SIGN IN SIGN UP

Fix GatedDeltaNet A_log dtype to prevent -inf under bfloat16 init (#47944)

* Fix GatedDeltaNet A_log dtype to prevent -inf under bfloat16 init

Sample A_log in float32 (instead of the module's default dtype) when
initializing Qwen3_5, Qwen3_5Moe, and Qwen3Next GatedDeltaNet modules.
Under bfloat16, torch.empty(...).uniform_(0, 16) can round a value to
0, and log(0) = -inf, permanently zeroing that head's decay gradient
for the rest of training.

Adds a regression test for each of the three affected models asserting
A_log stays float32 and finite when the model is constructed with
dtype=torch.bfloat16.

Fixes #47831

* Simplify A_log init fix: bump uniform_ lower bound instead of float32 cast

Per review feedback, avoid the -inf-on-log(0) bug by sampling A from uniform_(0.01, 16) instead of casting to float32. Drops the added regression test and dtype casting in favor of a one-line comment.
N
Nicholas Kluge committed
95940bf8775059a42f047256f076e4f607bc43ec
Parent: ed3edd8
Committed by GitHub <noreply@github.com> on 8/14/2026, 11:45:57 AM