Fix GatedDeltaNet A_log dtype to prevent -inf under bfloat16 init (#47944)
* Fix GatedDeltaNet A_log dtype to prevent -inf under bfloat16 init Sample A_log in float32 (instead of the module's default dtype) when initializing Qwen3_5, Qwen3_5Moe, and Qwen3Next GatedDeltaNet modules. Under bfloat16, torch.empty(...).uniform_(0, 16) can round a value to 0, and log(0) = -inf, permanently zeroing that head's decay gradient for the rest of training. Adds a regression test for each of the three affected models asserting A_log stays float32 and finite when the model is constructed with dtype=torch.bfloat16. Fixes #47831 * Simplify A_log init fix: bump uniform_ lower bound instead of float32 cast Per review feedback, avoid the -inf-on-log(0) bug by sampling A from uniform_(0.01, 16) instead of casting to float32. Drops the added regression test and dtype casting in favor of a one-line comment.
N
Nicholas Kluge committed
95940bf8775059a42f047256f076e4f607bc43ec
Parent: ed3edd8
Committed by GitHub <noreply@github.com>
on 8/14/2026, 11:45:57 AM