SIGN IN SIGN UP

FlashInfer: Kernel Library for LLM Serving

0 0 188 Python

feat: Reland support lse in trtllm paged attn kernels (#3116)

<!-- .github/pull_request_template.md -->

## ๐Ÿ“Œ Description

Re-land https://github.com/flashinfer-ai/flashinfer/pull/3058, after
revert https://github.com/flashinfer-ai/flashinfer/pull/3079. Issues
with original land investigated in
https://github.com/flashinfer-ai/flashinfer/issues/3114.

Re-introduce compute + output LSE support to paged context / decode /
mla operators.

To validate against the original versions issues, I ran
`compute-sanitizer --tool memcheck` on the previously failing
prefill/decode/MLA cases reports with 0 errors (prefill
`[...-256-32-2-8-HND]`), confirming the OOB is fixed.

Changes from original version:

**C++ changes**:
`include/flashinfer/trtllm/fmha/fmhaRunnerParams.h`: Added
`lseStrideTokens` and `lseStrideHeads` next to `lsePtr`.
`csrc/trtllm_fmha_kernel_launcher.cu`:
`trtllm_paged_attention_launcher` / `trtllm_ragged_attention_launcher`:
made `softmaxStatsPtr` allocation conditional on `lse != nullptr`, sized
it to the kernel's tile-aware layout `num_qo_heads * batch_size *
round_up(max_q_len, 256) * sizeof(float2)`, and plumbed LSE pointer +
strides into `runner_params`.
Kept the 8MB counter slab at a fixed offset in generation mode so the
existing workspace guard assertion still holds.
`trtllm_paged_attention_decode` / `trtllm_paged_attention_context` /
`trtllm_ragged_attention` wrappers now accept `Optional<TensorView> lse`
plus strides and forward them to the launcher.
**Python API**
`flashinfer/decode.py` (`trtllm_batch_decode_with_kv_cache`,
`TrtllmGenDecodeModule._paged_run`), `flashinfer/mla/_core.py`
(`trtllm_batch_decode_with_kv_cache_mla`), `flashinfer/prefill.py`
(`trtllm_batch_context_with_kv_cache`, `_paged_run in
get_trtllm_gen_prefill_module`): added `lse` / `return_lse` args with
shape `(num_tokens, num_qo_heads)` `float32` validation, passed strides
through.
XQA (and cute-dsl for MLA) backends raise `NotImplementedError` when
`return_lse=True` / `lse is not None`.

## ๐Ÿ” Related Issues

<!-- Link any related issues here -->

## ๐Ÿš€ Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### โœ… Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).

## ๐Ÿงช Tests

- [x] Tests have been added or updated as needed.
- [x] All tests are passing (`unittest`, etc.).

## Reviewer Notes

<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Optional LSE (Log-Sum-Exp) buffer support across paged/ragged
attention and public decode/context APIs; callers can provide or request
returned LSE (trtllm-gen only). Stride metadata is computed and
propagated; unsupported backends raise NotImplemented when LSE is
requested.

* **Bug Fixes / Stability**
* Runtime LSE validation (dtype/shape), conditional workspace
allocation/initialization to avoid unnecessary buffers, and guard-byte
protection to detect/prevent out-of-bounds writes.

* **Tests**
* End-to-end LSE tests validating dtype/shape/values and guard-region
overflow detection.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Alex Yang <aleozlx@gmail.com>
M
Matt Murphy committed
a73c281a78faecaf561c807126c2cefa49da914a
Parent: 719ee23
Committed by GitHub <noreply@github.com> on 5/16/2026, 5:25:41 AM