Extend autotuner delay kernel length (#3373)
<!-- .github/pull_request_template.md --> ## ๐ Description Increased the autotuner stream delay before timing individual tactics. For very small GEMMs, the previous delay was too short to fully absorb CPU enqueue overhead, so CUDA event timings could include GPU idle gaps while waiting for the host to submit subsequent work. A longer delay makes tactic profiling more representative of actual kernel execution time. 1000 us delay kernel Actual nsys time: 0.022 ms cuda_events: 0.021 ms cupti: 0.044 ms5000 us delay kernel Actual nsys time: 0.022 ms cuda_events: 0.022 ms cupti: 0.044 ms Comparing to result on RTX PRO 6000: 1000 us delay kernel Actual nsys time: 0.007 ms cuda_events: 0.023 ms cupti: 0.024 ms5000 us delay kernel Actual nsys time: 0.007 ms cuda_events: 0.006 ms cupti: 0.012 ms Conclusion is: The delay kernel length issue only affects the kernel with extremely short execution time (~10 us?), which makes sense to me, since with longer and longer kernel time, the cpu overhead is more and more negligible After using adequate delay kernel time, the remaining benchmark overhead issue is the same on both x86 and aarch64 machine, that cuda event can accurately reproduce the nsys observation. but there is still some unknown overhead for cupti path I also tried the same benchmark on B200, on which both cuda events and cupti can accurately reproduce the nsys time, no matter the delay kernel length ## ๐ Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### โ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## ๐งช Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Increased the default stream delay from 1 ms to 5 ms. * Documentation updated to reflect the new default. * No other tuning or runtime logic was changed; existing behavior remains compatible. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/flashinfer-ai/flashinfer/pull/3373?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Yanqin Zhai <yanqinz@nvidia.com>
Y
yanqinz2 committed
47cbd2cf6ed2c3f58db7f932252f51d9468fbc7b
Parent: 7ac0b5f
Committed by GitHub <noreply@github.com>
on 5/27/2026, 6:22:23 PM