ci: request a whole node for the vLLM/SGLang sanity allocation (#2007)
The sanity allocation asked for `--gres=gpu:2 --mincpus=24 --mem=128G`, copied from the mizu-based matrices. On dlcluster an allocation gets the whole node, so asking for a slice of a 4-GPU node is what let SLURM place a second job on it. The sanity servers are pinned to `CUDA_VISIBLE_DEVICES=0` and `1` while the enroot container is not device-constrained — `nvidia-smi -L` inside it lists all four cards — so both jobs drove the same two physical GPUs. The prefill server came up on a card another job had already filled (15.29 GB free of 186 GB, while its sibling card had 183.44 GB) and died at init with `RuntimeError: Not enough memory` (SGLang) or a free-memory `ValueError` (vLLM), after which the health check burned its full 300s on a process that was already gone. Seen on `nixl-ci-build-wheel` 1212, 1214 and 1215. - Drop `--gres`, `--mincpus` and `--mem` from the sanity allocation and pass only `--account`, matching the other two dlcluster jobs (`test-dl-matrix`, `test-dl-ep-matrix`). - Remove the now-unused `SLURM_GRES` / `SLURM_MEM` / `SLURM_MINCPUS` variables. - Update the sanity script comments that justified port handling, GPU memory fraction and process cleanup by several jobs sharing a node — that no longer happens, and the port comment also had the node at 8 GPUs rather than 4. Comments only, no behaviour change. The allocation then covers the node and nothing else lands on it, so the fixed device indices in `.gitlab/test_vllm_sglang_sanity.sh` are correct as they stand. Separately worth raising with the cluster admins: `ConstrainDevices=yes` in `cgroup.conf` would stop a container seeing GPUs outside its allocation at all, which is the general fix for this class of bug. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Simplified build environment resource allocation settings by adjusting SLURM-related parameters used during wheel builds. * Updated internal test script comments to clarify dynamic port selection and related process/handshake behavior when running multiple server components concurrently. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: NirWolfer <nwolfer@nvidia.com>
N
NirWolfer committed
ad661a7212170db72e7c2505b4f9faaada1dc533
Parent: 9ed1f01
Committed by GitHub <noreply@github.com>
on 8/2/2026, 6:58:41 AM