SIGN IN SIGN UP

FlashInfer: Kernel Library for LLM Serving

0 0 188 Python

feat: Support 4over6 nvfp4 for quantizer and fused MoE (#3264)

<!-- .github/pull_request_template.md -->

## ๐Ÿ“Œ Description
@humansand

Implement 4over6 nvfp4 from:
- Paper: https://arxiv.org/abs/2512.02010
- Code: https://github.com/mit-han-lab/fouroversix

TE PR: 
- https://github.com/NVIDIA/TransformerEngine/pull/2972

For original nvfp4, only `cutlass_fused_moe` is supported.
For per-token nvfp4, only `trtllm_fp4_block_scale_routed_moe` and
`trtllm_fp4_block_scale_moe` are supported.

The results is bitwise exact with reference implementation by enabling:
- `TRTLLM_DISABLE_FP4_QUANT_FAST_MATH=1`

Under strict no fast math mode, the quantizer is bitwise exact with
pytorch reference implementation.

By default, 448 scaling is used for E4M3. To enable the 256/384 E4M3
scaling in the original paper, enable
`FLASHINFER_NVFP4_4OVER6_E4M3_USE_256`.

Error mode can be either `MAE` or `MSE`, default to `MAE`.

Need to rebase after:
- https://github.com/flashinfer-ai/flashinfer/pull/3237
- https://github.com/flashinfer-ai/flashinfer/pull/3027

Future work:
- TE recipe implementation after
https://github.com/NVIDIA/TransformerEngine/pull/2931 is merged
- Performance optimization
<!-- What does this PR do? Briefly describe the changes and why theyโ€™re
needed. -->

## ๐Ÿ” Related Issues

<!-- Link any related issues here -->

## ๐Ÿš€ Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### โœ… Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).

## ๐Ÿงช Tests

- [x] Tests have been added or updated as needed.
- [x] All tests are passing (`unittest`, etc.).

## Reviewer Notes

<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Release Notes

* **New Features**
* Introduced NVFP4 "4-over-6" quantization mode for improved FP4
precision, configurable via environment variables
* Added MSE-based scale candidate selection to enhance quantization
accuracy
* Implemented runtime toggles for FP4 fast-math and optimization control

* **Improvements**
* Enhanced FP4 quantization kernel dispatch for flexible runtime
configuration

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/flashinfer-ai/flashinfer/pull/3264)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Lain <siyuanf@nvidia.com>
Z
Ziang Li committed
a0f61ad36601b2fdadf0840dbc41ade9be46ed01
Parent: 209c16a
Committed by GitHub <noreply@github.com> on 5/21/2026, 6:24:42 AM