non-override tactic control (#3260)
<!-- .github/pull_request_template.md --> ## 📌 Description It also fixes a tactic mismatch hazard in the non-override path. Autotuning buckets dynamic M values, but static cuDNN graphs are rebuilt with the actual runtime M. A tactic index profiled on a bucket-M graph may not refer to the same execution plan in the actual-M graph. To avoid applying an invalid or mismatched tactic, the non-override cuDNN paths now expose only the fallback tactic -1, forcing runtime to use the cuDNN heuristic path for the actual static graph. This update also brings the cuDNN FP8 and MXFP8 GEMM paths in line with the existing BF16/FP4 override-shape design. For FP8, the per-tensor quantized cuDNN graph builders were renamed to follow the same naming convention as the other GEMM paths: build_cudnn_gemm_fp8_graph and build_cudnn_gemm_fp8_graph_override_shape. The cuDNN FP8 runner now supports the override-shape execution path, using the autotuner’s effective M-bucket mapper to build a reusable bucketed graph and passing the runtime shapes through override_shapes. For MXFP8, the cuDNN graph construction was refactored to match the FP4/BF16 structure. The previous create_cudnn_execution_plans_mxfp8_gemm + _get_cudnn_mxfp8_gemm_graph split was replaced with a single build_cudnn_gemm_mxfp8_graph builder that owns graph creation, support checking, and plan building. The MXFP8 runner now also supports the override-shape path with bucketed graph reuse. For both FP8 and MXFP8, the non-override static cuDNN path no longer participates in autotune plan-index profiling, since static graphs are rebuilt for the actual runtime M and cannot safely reuse tactic indices profiled on bucketed M shapes. ## 🔍 Related Issues test_bmm_bf16 failure on B300 ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **API Changes** * Removed legacy cuDNN GEMM override helpers from the public API; core GEMM wrappers remain. * Public exports now include available CuTe‑DSL kernels at import time when present. * **Improvements** * Safer override‑shape handling with deterministic fallback when dynamic-shape support is unavailable. * Improved cache invalidation to avoid stale execution plans. * **Tests** * Added GPU tests validating override‑shape execution across dynamic sizes and quantized modes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Yanqin Zhai <yanqinz@nvidia.com>
Y
yanqinz2 committed
f6717ff6bc6061c4eb0474576746ee1b42bd6325
Parent: 059008c
Committed by GitHub <noreply@github.com>
on 5/8/2026, 10:13:39 PM