SIGN IN SIGN UP

ci: request a whole node for the vLLM/SGLang sanity allocation (#2007)

The sanity allocation asked for `--gres=gpu:2 --mincpus=24 --mem=128G`,
copied from the mizu-based matrices. On dlcluster an allocation gets the
whole node, so asking for a slice of a 4-GPU node is what let SLURM
place a second job on it.

The sanity servers are pinned to `CUDA_VISIBLE_DEVICES=0` and `1` while
the enroot container is not device-constrained — `nvidia-smi -L` inside
it lists all four cards — so both jobs drove the same two physical GPUs.
The prefill server came up on a card another job had already filled
(15.29 GB free of 186 GB, while its sibling card had 183.44 GB) and died
at init with `RuntimeError: Not enough memory` (SGLang) or a free-memory
`ValueError` (vLLM), after which the health check burned its full 300s
on a process that was already gone. Seen on `nixl-ci-build-wheel` 1212,
1214 and 1215.

- Drop `--gres`, `--mincpus` and `--mem` from the sanity allocation and
pass only `--account`, matching the other two dlcluster jobs
(`test-dl-matrix`, `test-dl-ep-matrix`).
- Remove the now-unused `SLURM_GRES` / `SLURM_MEM` / `SLURM_MINCPUS`
variables.
- Update the sanity script comments that justified port handling, GPU
memory fraction and process cleanup by several jobs sharing a node —
that no longer happens, and the port comment also had the node at 8 GPUs
rather than 4. Comments only, no behaviour change.

The allocation then covers the node and nothing else lands on it, so the
fixed device indices in `.gitlab/test_vllm_sglang_sanity.sh` are correct
as they stand.

Separately worth raising with the cluster admins: `ConstrainDevices=yes`
in `cgroup.conf` would stop a container seeing GPUs outside its
allocation at all, which is the general fix for this class of bug.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Simplified build environment resource allocation settings by adjusting
SLURM-related parameters used during wheel builds.
* Updated internal test script comments to clarify dynamic port
selection and related process/handshake behavior when running multiple
server components concurrently.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: NirWolfer <nwolfer@nvidia.com>
N
NirWolfer committed
ad661a7212170db72e7c2505b4f9faaada1dc533
Parent: 9ed1f01
Committed by GitHub <noreply@github.com> on 8/2/2026, 6:58:41 AM