SIGN IN SIGN UP

FlashInfer: Kernel Library for LLM Serving

0 0 188 Python

non-override tactic control (#3260)

<!-- .github/pull_request_template.md -->

## 📌 Description

It also fixes a tactic mismatch hazard in the non-override path.
Autotuning buckets dynamic M values, but static cuDNN graphs are rebuilt
with the actual runtime M. A tactic index profiled on a bucket-M graph
may not refer to the same execution plan in the actual-M graph. To avoid
applying an invalid or mismatched tactic, the non-override cuDNN paths
now expose only the fallback tactic -1, forcing runtime to use the cuDNN
heuristic path for the actual static graph.

This update also brings the cuDNN FP8 and MXFP8 GEMM paths in line with
the existing BF16/FP4 override-shape design.

For FP8, the per-tensor quantized cuDNN graph builders were renamed to
follow the same naming convention as the other GEMM paths:
build_cudnn_gemm_fp8_graph and
build_cudnn_gemm_fp8_graph_override_shape. The cuDNN FP8 runner now
supports the override-shape execution path, using the autotuner’s
effective M-bucket mapper to build a reusable bucketed graph and passing
the runtime shapes through override_shapes.

For MXFP8, the cuDNN graph construction was refactored to match the
FP4/BF16 structure. The previous create_cudnn_execution_plans_mxfp8_gemm
+ _get_cudnn_mxfp8_gemm_graph split was replaced with a single
build_cudnn_gemm_mxfp8_graph builder that owns graph creation, support
checking, and plan building. The MXFP8 runner now also supports the
override-shape path with bucketed graph reuse.

For both FP8 and MXFP8, the non-override static cuDNN path no longer
participates in autotune plan-index profiling, since static graphs are
rebuilt for the actual runtime M and cannot safely reuse tactic indices
profiled on bucketed M shapes.

## 🔍 Related Issues

test_bmm_bf16 failure on B300

## 🚀 Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### ✅ Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).

## 🧪 Tests

- [ ] Tests have been added or updated as needed.
- [ ] All tests are passing (`unittest`, etc.).

## Reviewer Notes

<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **API Changes**
* Removed legacy cuDNN GEMM override helpers from the public API; core
GEMM wrappers remain.
* Public exports now include available CuTe‑DSL kernels at import time
when present.

* **Improvements**
* Safer override‑shape handling with deterministic fallback when
dynamic-shape support is unavailable.
  * Improved cache invalidation to avoid stale execution plans.

* **Tests**
* Added GPU tests validating override‑shape execution across dynamic
sizes and quantized modes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Yanqin Zhai <yanqinz@nvidia.com>
Y
yanqinz2 committed
f6717ff6bc6061c4eb0474576746ee1b42bd6325
Parent: 059008c
Committed by GitHub <noreply@github.com> on 5/8/2026, 10:13:39 PM