MLA Decode Autotuning Across TRTLLM-Gen and CuTe Backends (#3355)
<!-- .github/pull_request_template.md --> ## ๐ Description <!-- What does this PR do? Briefly describe the changes and why theyโre needed. --> This PR enables autotuning between the "trtllm-gen" and "cute-dsl" backends in the `trtllm_batch_decode_with_kv_cache_mla` API when running with `backend="auto"` and `flashinfer.autotune(true)`. ## Autotuner Performance Results With a cold kernel cache, each layer takes ~5 minutes to autotune. Otherwise with a warm kernel cache, autotuning the layer takes 5 seconds. The following table contains the speedups of running `trtllm_batch_decode_with_kv_cache_mla` with `backend="auto"` and `flashinfer.autotune(true)` before vs. after this PR on a B200 machine. | batch \ s_kv | 512 | 1024 | 2048 | 4096 | 8192 | |---:|---:|---:|---:|---:|---:| | **1** | 1.17ร | 1.31ร | 1.21ร | 1.44ร | 1.50ร | | **2** | 1.17ร | 1.31ร | 1.27ร | 1.29ร | 1.60ร | | **4** | 1.15ร | 1.36ร | 1.38ร | **1.63ร** | 1.17ร | | **8** | 1.14ร | 1.44ร | **1.68ร** | 1.17ร | 1.12ร | | **16** | 1.12ร | **1.53ร** | 1.08ร | 1.06ร | 1.06ร | | **32** | 1.00ร | 1.04ร | 1.06ร | 1.06ร | 1.04ร | | **64** | 1.14ร | 1.10ร | 1.08ร | 1.08ร | 1.07ร | | **128** | 0.95ร | 1.00ร | 1.02ร | 1.02ร | 1.00ร | | **256** | 1.00ร | 1.00ร | 1.00ร | 1.00ร | 0.99ร | This aligns with expectations, as previously we found the "cute-dsl" backend to be faster for smaller batch sizes. ## ๐ Related Issues <!-- Link any related issues here --> This is requested in [issue 2891](https://github.com/flashinfer-ai/flashinfer/issues/2891#event-24551929489). ## ๐ Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### โ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## ๐งช Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a --autotune CLI to the MLA paged-attention benchmark to run warmup/autotune passes before timing. * Enabled autotuning for MLA batch decode with automatic backend dispatch and dynamic batch tuning; expanded backend availability on select hardware. * **Tests** * Added end-to-end autotune smoke tests for MLA decode using the auto backend. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/flashinfer-ai/flashinfer/pull/3355?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
V
Vincent committed
0b755de7060980b6ced02359aac6921aef5418d2
Parent: 96ab2ce
Committed by GitHub <noreply@github.com>
on 5/26/2026, 9:18:19 PM