SIGN IN SIGN UP

FlashInfer: Kernel Library for LLM Serving

0 0 188 Python

fix(autotuner): differentiate file cache entries by runner specific kernel parameters (#3367)

## ๐Ÿ“Œ Description

The persistent autotune file cache key was constructed as a 3-tuple
`(custom_op, runner_class, profile)`, intentionally dropping
`hash(runner)` for cross-process stability, but **unintentionally also
dropping `extras`**, which carries runnerspecific parameters like
`use_8x4_sf_layout`, which caused `TrtllmGemmRunner` instances with
`use_8x4_sf_layout=True` and `use_8x4_sf_layout=False` to collide in the
file cache. When vLLM or other frameworks persists and reloads autotune
results, the wrong tactic gets applied, producing:
```
RuntimeError: Check failed: (config.mOptions.mSfLayoutB == mOptions.sfLayoutB)
             is false: Invalid sf layout in run
```

**Updates:**

1. **`flashinfer/autotuner.py`**, extend file key from 3-tuple to
4-tuple in `search_cache`, `save_configs`, `load_from_file`.
2. **`flashinfer/gemm/gemm_base.py`**, implement
`get_cache_key_extras()` to return `(self._use_8x4_sf_layout,)`.
3. **`tests/autotuner/test_autotuner_configs.py`**, add
`TestFileCacheKeyCollision` with two tests reproducing and update
existing tests for the new 4-tuple key format.

## ๐Ÿ” Related Issues

- Fixes flashinfer-ai/flashinfer#3363
- Related to vllm-project/vllm#42537

## ๐Ÿš€ Pull Request Checklist

### โœ… Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

## ๐Ÿงช Tests

- [x] Tests have been added or updated as needed.
- [x] All tests are passing (`unittest`, etc.).
  - `test_autotuner_configs.py` โ€” **37 passed**
  - `test_autotuner_core.py` โ€” **78 passed**

## Reviewer Notes

The bug is latent until autotune results are persisted to disk and
reloaded. It was exposed by vllm-project/vllm#42537 which added
persistent caching for FlashInfer autotuning in vLLM.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Fixed cache key mismatch between memory and disk so runner-specific
parameters (extras/layout) are preserved when saving/loading autotuner
results.
* Resolved collisions where configurations that differed only by
layout-related parameters were treated as the same.

* **Tests**
* Added and updated tests to verify distinct file-cache entries and
loading for entries differing by extras/layout.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/flashinfer-ai/flashinfer/pull/3367?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Albert Cheng (Engrg-Hardware 1) <albecheng@login-lyris01.lyris.clusters.nvidia.com>
A
Albert Cheng committed
2f372e2e5f7c1c894c6aed11e4f02fc6dd69ee56
Parent: dac8358
Committed by GitHub <noreply@github.com> on 5/22/2026, 8:22:46 PM