feat: Support 4over6 nvfp4 for quantizer and fused MoE (#3264)
<!-- .github/pull_request_template.md --> ## ๐ Description @humansand Implement 4over6 nvfp4 from: - Paper: https://arxiv.org/abs/2512.02010 - Code: https://github.com/mit-han-lab/fouroversix TE PR: - https://github.com/NVIDIA/TransformerEngine/pull/2972 For original nvfp4, only `cutlass_fused_moe` is supported. For per-token nvfp4, only `trtllm_fp4_block_scale_routed_moe` and `trtllm_fp4_block_scale_moe` are supported. The results is bitwise exact with reference implementation by enabling: - `TRTLLM_DISABLE_FP4_QUANT_FAST_MATH=1` Under strict no fast math mode, the quantizer is bitwise exact with pytorch reference implementation. By default, 448 scaling is used for E4M3. To enable the 256/384 E4M3 scaling in the original paper, enable `FLASHINFER_NVFP4_4OVER6_E4M3_USE_256`. Error mode can be either `MAE` or `MSE`, default to `MAE`. Need to rebase after: - https://github.com/flashinfer-ai/flashinfer/pull/3237 - https://github.com/flashinfer-ai/flashinfer/pull/3027 Future work: - TE recipe implementation after https://github.com/NVIDIA/TransformerEngine/pull/2931 is merged - Performance optimization <!-- What does this PR do? Briefly describe the changes and why theyโre needed. --> ## ๐ Related Issues <!-- Link any related issues here --> ## ๐ Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### โ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## ๐งช Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Introduced NVFP4 "4-over-6" quantization mode for improved FP4 precision, configurable via environment variables * Added MSE-based scale candidate selection to enhance quantization accuracy * Implemented runtime toggles for FP4 fast-math and optimization control * **Improvements** * Enhanced FP4 quantization kernel dispatch for flexible runtime configuration [](https://app.coderabbit.ai/change-stack/flashinfer-ai/flashinfer/pull/3264) <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Lain <siyuanf@nvidia.com>
Z
Ziang Li committed
a0f61ad36601b2fdadf0840dbc41ade9be46ed01
Parent: 209c16a
Committed by GitHub <noreply@github.com>
on 5/21/2026, 6:24:42 AM