SIGN IN SIGN UP

UCT/IB/GDAKI: Use NVML to enumerate GPUs in dev_matrix_init (#11460)

* UCT/IB/GDAKI: Use NVML to enumerate GPUs in dev_matrix_init

CUDA_VISIBLE_DEVICES hides physical GPUs from cuDeviceGetCount(), which
made uct_gdaki_dev_matrix_init() compute the wrong NIC/GPU mapping and
ib_per_cuda ratio. Enumerate GPUs through NVML (which ignores the
visibility mask) for topology purposes, keep cuda_map storing CUDA
driver indices, and skip setting the bit for hidden GPUs. Fall back to
the CUDA-only enumeration when NVML is unavailable.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* UCT/IB/GDAKI: Use NVML to enumerate GPUs in dev_matrix_init - 2

* UCT/IB/GDAKI: Use NVML to enumerate GPUs in dev_matrix_init - 3

* UCT/IB/GDAKI: Use NVML to enumerate GPUs in dev_matrix_init - 4

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A
Artemy Kovalyov committed
9e28a6954bdd80710dfd37f1435e674d06fa9eab
Parent: 2faf305
Committed by GitHub <noreply@github.com> on 5/26/2026, 4:17:26 PM