SIGN IN SIGN UP

nixl_ep: Safely connect ranks during traffic (#2138)

connect_ranks(activate=False) previously destroyed and recreated active
memory views while dispatch/combine kernels could still be using them.
This made connecting new ranks during traffic unsafe.

Also, when the CUDA IPC cache is disabled, recreating memory views
closes and reopens all IPC handles, which is slow.

Instead, during async connect_ranks, stage standby memory views while
keeping the active views alive, then commit them when the new ranks are
activated. This keeps inflight dispatch/combine kernels safe and avoids
unnecessarily reopening existing IPC handles.

Also, release the Python GIL while connecting new ranks to avoid
stalling request serving on the main inference engine thread.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Bug Fixes

* Improved memory resource handling during device connections,
disconnections, and configuration updates.
* Refreshed device context and peer-to-peer memory references more
reliably when memory settings change.
* Ensured both active and pending memory resources are released during
cleanup.
* Improved connection setup responsiveness by reducing unnecessary
blocking during initialization.

## Documentation

* Clarified that connections can be established concurrently with
dispatch and combine operations, but not during topology or lifecycle
changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Signed-off-by: Linoy Geva <lgeva@nvidia.com>
Co-authored-by: Linoy Geva <lgeva@nvidia.com>
I
Itay Alroy committed
b0cbb237354d72b83500d5214d2b4484f9866fa3
Parent: 885c49d
Committed by GitHub <noreply@github.com> on 8/25/2026, 5:36:59 PM