SIGN IN SIGN UP

FlashInfer: Kernel Library for LLM Serving

0 0 188 Python

MLA Decode Autotuning Across TRTLLM-Gen and CuTe Backends (#3355)

<!-- .github/pull_request_template.md -->

## ๐Ÿ“Œ Description

<!-- What does this PR do? Briefly describe the changes and why theyโ€™re
needed. -->

This PR enables autotuning between the "trtllm-gen" and "cute-dsl"
backends in the `trtllm_batch_decode_with_kv_cache_mla` API when running
with `backend="auto"` and `flashinfer.autotune(true)`.

## Autotuner Performance Results

With a cold kernel cache, each layer takes ~5 minutes to autotune.
Otherwise with a warm kernel cache, autotuning the layer takes 5
seconds.

The following table contains the speedups of running
`trtllm_batch_decode_with_kv_cache_mla` with `backend="auto"` and
`flashinfer.autotune(true)` before vs. after this PR on a B200 machine.

  | batch \ s_kv | 512 | 1024 | 2048 | 4096 | 8192 |
  |---:|---:|---:|---:|---:|---:|
  | **1** | 1.17ร— | 1.31ร— | 1.21ร— | 1.44ร— | 1.50ร— |
  | **2** | 1.17ร— | 1.31ร— | 1.27ร— | 1.29ร— | 1.60ร— |
  | **4** | 1.15ร— | 1.36ร— | 1.38ร— | **1.63ร—** | 1.17ร— |
  | **8** | 1.14ร— | 1.44ร— | **1.68ร—** | 1.17ร— | 1.12ร— |
  | **16** | 1.12ร— | **1.53ร—** | 1.08ร— | 1.06ร— | 1.06ร— |
  | **32** | 1.00ร— | 1.04ร— | 1.06ร— | 1.06ร— | 1.04ร— |
  | **64** | 1.14ร— | 1.10ร— | 1.08ร— | 1.08ร— | 1.07ร— |
  | **128** | 0.95ร— | 1.00ร— | 1.02ร— | 1.02ร— | 1.00ร— |
  | **256** | 1.00ร— | 1.00ร— | 1.00ร— | 1.00ร— | 0.99ร— |

This aligns with expectations, as previously we found the "cute-dsl"
backend to be faster for smaller batch sizes.

## ๐Ÿ” Related Issues

<!-- Link any related issues here -->

This is requested in [issue
2891](https://github.com/flashinfer-ai/flashinfer/issues/2891#event-24551929489).

## ๐Ÿš€ Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### โœ… Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).

## ๐Ÿงช Tests

- [x] Tests have been added or updated as needed.
- [x] All tests are passing (`unittest`, etc.).

## Reviewer Notes

<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a --autotune CLI to the MLA paged-attention benchmark to run
warmup/autotune passes before timing.
* Enabled autotuning for MLA batch decode with automatic backend
dispatch and dynamic batch tuning; expanded backend availability on
select hardware.

* **Tests**
* Added end-to-end autotune smoke tests for MLA decode using the auto
backend.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/flashinfer-ai/flashinfer/pull/3355?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
V
Vincent committed
0b755de7060980b6ced02359aac6921aef5418d2
Parent: 96ab2ce
Committed by GitHub <noreply@github.com> on 5/26/2026, 9:18:19 PM