SIGN IN SIGN UP

FlashInfer: Kernel Library for LLM Serving

0 0 188 Python

feat: MNNVL Allreduce quant fusion and performance optimization (#3385)

<!-- .github/pull_request_template.md -->

## ๐Ÿ“Œ Description
- Use named barrier and cluster barrier instead of a sync.
- Avoid loading local buffer again in oneshot, use template-based fash
path for world size <= 8.
- Adjust grid dispatch policy to use more SMs at the cost of single SM
occupancy for small batch sizes.
- Extended allreduce_fusion so MNNVL supports standard FP8/NVFP4 quant
patterns while keeping MoE and packed-group quant paths TRTLLM-only.
(replace #2263 )
- Added Hopper/Blackwell-only JIT arch gating.
- Added focused correctness coverage for quant fusion across dtype,
strategy, layout, shape, and norm-output variants, plus NVFP4 validation
for padded swizzled scale buffers.

**Performance Change Dashboard:
[report.html](https://github.com/user-attachments/files/28164512/report.html)**

For M <= 8, fused oneshot is 6.24% faster latency-weighted and ar_only
oneshot is 4.26% faster. The main benefit is from avoiding loading the
local buffer in lamport polling.
Fused two-shot gets benefit for relatively larger batch size. The main
benefit is from reducing CTA sync overhead by using named barrier
instead of syncthread.
 
## ๐Ÿ” Related Issues

None
## ๐Ÿš€ Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### โœ… Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).

## ๐Ÿงช Tests

- [x] Tests have been added or updated as needed.
- [x] All tests are passing (`unittest`, etc.).

## Reviewer Notes

<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* FP8 and NVFP4 quantized AllReduce + residual-add + RMSNorm fusion with
optional quant outputs, scale outputs, and configurable scale layouts;
expanded fusion patterns and execution strategies (oneshot/twoshot).

* **Behavior / Validation**
* Stricter shape/dtype/layout checks, default swizzled layout, explicit
errors for unsupported quant/layout/CUDA combos, and enforced coupling
of quantization with RMSNorm and scale tensors.

* **Documentation**
* Updated API docs and examples describing quantization patterns,
layouts, and strategy/reduction-order behavior.

* **Tests**
* End-to-end FP8/NVFP4 tests added, including negative tests validating
invalid NVFP4/scale cases.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/flashinfer-ai/flashinfer/pull/3385?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Shiyu Li <shili@nvl72d081-T12.cm.cluster>
S
Shiyu Li committed
d53f10682cfdc13c12d3463b2559355b9eb1671d
Parent: 56d537a
Committed by GitHub <noreply@github.com> on 5/26/2026, 5:24:42 PM