Correct the fp16 guard: the rejection sampler is not bf16-only
I wrote that fp16 fails in two places and only checked one. rejection_sampler_utils.py contains zero bf16/fp16 literals -- every cast is tl.float32 or tl.int64 and its buffers are allocated float32/int64 off target_logits.new_empty, so it is dtype-agnostic. The device-side assert @ahnguyen17 saw in rejection_sample() with SPEC_ATTN=0 has some other cause; it was on a w8a16 GPTQ target whose GEMM path was already failing three other ways, and a device-side assert reads like an out-of-bounds index rather than a dtype mismatch. Not established, so no longer claimed. The guard stands and the refusal is unchanged -- the split-KV verify kernel really does hardcode tl.bfloat16 at spec-decode-attn.patch:217,236, which is a compile failure at the first attention, and fp16 with a speculator is untested here either way. Only the stated reason shrinks to what I verified.
M
mhenrichsen committed
6635cd85786cf9b9aa50f488ddb4d5c1f39fbcdc
Parent: 50b6888