SIGN IN SIGN UP

FlashInfer: Kernel Library for LLM Serving

0 0 188 Python

fix(gdn): allow importing gdn_decode without a CUDA device (#3293)

<!-- .github/pull_request_template.md -->

## ๐Ÿ“Œ Description

- Defer `torch.cuda.get_device_properties()` and
`torch.cuda.get_device_capability()` in `gdn_decode_bf16_state.py `from
module scope to lazy `@functools.cache` helpers that derive the device
from the input tensor (q.device), matching the convention used by the
CuTe DSL MLA and MoE backends.
- Catch RuntimeError in addition to ImportError in all gdn_decode.py
backend import blocks, matching the existing pattern in
`gdn_kernels/__init__.py.`

Before this PR: 
```
CUDA_VISIBLE_DEVICES= python3 -c "import flashinfer.gdn_decode; print('Success: import worked without CUDA devices')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/workspace/flashinfer/flashinfer/gdn_decode.py", line 58, in <module>
    from .gdn_kernels.gdn_decode_bf16_state import (
  File "/workspace/flashinfer/flashinfer/gdn_kernels/gdn_decode_bf16_state.py", line 1389, in <module>
    NUM_SMS = torch.cuda.get_device_properties(0).multi_processor_count
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py312/lib/python3.12/site-packages/torch/cuda/__init__.py", line 614, in get_device_properties
    _lazy_init()  # will define _get_device_properties
    ^^^^^^^^^^^^
  File "/opt/conda/envs/py312/lib/python3.12/site-packages/torch/cuda/__init__.py", line 410, in _lazy_init
    torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available
```

After this PR: 
```
CUDA_VISIBLE_DEVICES= python3 -c "import flashinfer.gdn_decode; print('Success: import worked without CUDA devices')"
Success: import worked without CUDA devices
```

## ๐Ÿ” Related Issues

https://github.com/flashinfer-ai/flashinfer/issues/3262

## ๐Ÿš€ Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### โœ… Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).

## ๐Ÿงช Tests

- [x] Tests have been added or updated as needed.
- [x] All tests are passing (`unittest`, etc.).

## Reviewer Notes

<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Decode backends now treat runtime import failures like import errors
and gracefully fall back to safe defaults.

* **Performance / Reliability**
* GPU kernels adapt at runtime to the actual device, selecting
device-aware configurations and maintaining separate compiled kernels
per device for more reliable and optimized execution.

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/flashinfer-ai/flashinfer/pull/3293)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
K
Ka-Hyun Nam committed
885d0cf4803f99a04574aa7777a74d50dd713e20
Parent: 89bfb68
Committed by GitHub <noreply@github.com> on 5/14/2026, 9:53:56 PM