feat: MNNVL Allreduce quant fusion and performance optimization (#3385)
<!-- .github/pull_request_template.md --> ## ๐ Description - Use named barrier and cluster barrier instead of a sync. - Avoid loading local buffer again in oneshot, use template-based fash path for world size <= 8. - Adjust grid dispatch policy to use more SMs at the cost of single SM occupancy for small batch sizes. - Extended allreduce_fusion so MNNVL supports standard FP8/NVFP4 quant patterns while keeping MoE and packed-group quant paths TRTLLM-only. (replace #2263 ) - Added Hopper/Blackwell-only JIT arch gating. - Added focused correctness coverage for quant fusion across dtype, strategy, layout, shape, and norm-output variants, plus NVFP4 validation for padded swizzled scale buffers. **Performance Change Dashboard: [report.html](https://github.com/user-attachments/files/28164512/report.html)** For M <= 8, fused oneshot is 6.24% faster latency-weighted and ar_only oneshot is 4.26% faster. The main benefit is from avoiding loading the local buffer in lamport polling. Fused two-shot gets benefit for relatively larger batch size. The main benefit is from reducing CTA sync overhead by using named barrier instead of syncthread. ## ๐ Related Issues None ## ๐ Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### โ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## ๐งช Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * FP8 and NVFP4 quantized AllReduce + residual-add + RMSNorm fusion with optional quant outputs, scale outputs, and configurable scale layouts; expanded fusion patterns and execution strategies (oneshot/twoshot). * **Behavior / Validation** * Stricter shape/dtype/layout checks, default swizzled layout, explicit errors for unsupported quant/layout/CUDA combos, and enforced coupling of quantization with RMSNorm and scale tensors. * **Documentation** * Updated API docs and examples describing quantization patterns, layouts, and strategy/reduction-order behavior. * **Tests** * End-to-end FP8/NVFP4 tests added, including negative tests validating invalid NVFP4/scale cases. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/flashinfer-ai/flashinfer/pull/3385?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Shiyu Li <shili@nvl72d081-T12.cm.cluster>
S
Shiyu Li committed
d53f10682cfdc13c12d3463b2559355b9eb1671d
Parent: 56d537a
Committed by GitHub <noreply@github.com>
on 5/26/2026, 5:24:42 PM