SIGN IN SIGN UP

FlashInfer: Kernel Library for LLM Serving

0 0 188 Python

Extend autotuner delay kernel length (#3373)

<!-- .github/pull_request_template.md -->

## ๐Ÿ“Œ Description

Increased the autotuner stream delay before timing individual tactics.
For very small GEMMs, the previous delay was too short to fully absorb
CPU enqueue overhead, so CUDA event timings could include GPU idle gaps
while waiting for the host to submit subsequent work. A longer delay
makes tactic profiling more representative of actual kernel execution
time.

1000 us delay kernel
Actual nsys time: 0.022 ms
cuda_events:      0.021 ms
cupti:            0.044 ms5000 us delay kernel
Actual nsys time: 0.022 ms
cuda_events:      0.022 ms
cupti:            0.044 ms

Comparing to result on RTX PRO 6000:
1000 us delay kernel
Actual nsys time: 0.007 ms
cuda_events:      0.023 ms
cupti:            0.024 ms5000 us delay kernel
Actual nsys time: 0.007 ms
cuda_events:      0.006 ms
cupti:            0.012 ms

Conclusion is:
The delay kernel length issue only affects the kernel with extremely
short execution time (~10 us?), which makes sense to me, since with
longer and longer kernel time, the cpu overhead is more and more
negligible
After using adequate delay kernel time, the remaining benchmark overhead
issue is the same on both x86 and aarch64 machine, that cuda event can
accurately reproduce the nsys observation. but there is still some
unknown overhead for cupti path
I also tried the same benchmark on B200, on which both cuda events and
cupti can accurately reproduce the nsys time, no matter the delay kernel
length

## ๐Ÿš€ Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### โœ… Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).

## ๐Ÿงช Tests

- [ ] Tests have been added or updated as needed.
- [ ] All tests are passing (`unittest`, etc.).

## Reviewer Notes

<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
  * Increased the default stream delay from 1 ms to 5 ms.
  * Documentation updated to reflect the new default.
* No other tuning or runtime logic was changed; existing behavior
remains compatible.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/flashinfer-ai/flashinfer/pull/3373?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Yanqin Zhai <yanqinz@nvidia.com>
Y
yanqinz2 committed
47cbd2cf6ed2c3f58db7f932252f51d9468fbc7b
Parent: 7ac0b5f
Committed by GitHub <noreply@github.com> on 5/27/2026, 6:22:23 PM