SIGN IN SIGN UP

build: derive CUDA_VERSION from the base image tag again (#2132)

## What?

Fix `CUDA_VERSION` resolution in `contrib/build-container.sh` so it is
derived from `--base-image-tag` again.

- Seed `CUDA_VERSION` empty and apply the `13.2` default *after* the
`BASE_IMAGE_TAG` derivation, so precedence is `--cuda-version` -> base
image tag -> `13.2`.
- Error out when the resolved CUDA major disagrees with
`--base-image-tag`.
- Update the `--cuda-version` help text, which interpolated a default
that no longer exists at parse time.

## Why?

The nightly cu12 wheel build fails in the OBJ plugin check
(`nixl-ci-build-wheel-nightly`, cu13 passes):

```
RUN ... if [ "${CUDA_VERSION%%.*}" = "13" ]; then
        OBJ_PLUGIN="$(find /usr/local/nixl -name libplugin_OBJ.so -print -quit)";
        readelf -d "$OBJ_PLUGIN" | grep 'libcuobjclient'; fi
subprocess exited with status 1
```

That branch is CUDA 13 only, but the cu12 build entered it.
`CUDA_VERSION` was seeded with a `13.2` default at the top of the script
(#1971), so the `BASE_IMAGE_TAG` derivation added later (#2110) was
written as `${CUDA_VERSION:-...}` and could never fire — every build
reported `13.2` regardless of its base image:

```
BEFORE, cu12 nightly (--base-image-tag 12.9.1-devel-ubi8) -> CUDA_VERSION=13.2
AFTER                                                     -> CUDA_VERSION=12.9
```

On a CUDA 12 base, `13.2` also names the toolkit symlink
`/usr/local/cuda-13.2` (`Dockerfile.manylinux:59`), so the `cuobjclient`
`.pc` `cudaroot` never resolves, meson does not link the OBJ plugin
against `libcuobjclient`, and the `grep` exits 1. The log shows no
`libplugin_OBJ.so not found`, so the plugin was built — only the link
was missing.

**Second effect, silent:** the meta-wheel copy at
`Dockerfile.manylinux:544` is gated on the CUDA major being `12`. With
both variants reporting `13.2` it fired on neither, so
`nixl-<ver>-py3-none-any.whl` has not been produced since #1971.
`nixl-ci-build-llm-container` fetches that exact filename with `curl
-f`. This change restores it via the cu12 build.

## How?

Verified against the real script with `docker` stubbed, reading the
`--build-arg CUDA_VERSION` it emits:

| invocation | resolved |
|---|---|
| manylinux, base `12.9.1-devel-ubi8`, no override | `12.9` (was `13.2`)
|
| manylinux, base `13.0.1-devel-ubi8`, no override | `13.0` |
| manylinux + `--cuda-version 12.6` | `12.6` |
| default `contrib/Dockerfile` (non-manylinux) | `13.2` (unchanged) |

The guard on a real mismatch:

```
$ ./contrib/build-container.sh --dockerfile contrib/Dockerfile.manylinux \
    --base-image-tag 12.9.1-devel-ubi8 --cuda-version 13.2
ERROR: CUDA_VERSION=13.2 disagrees with --base-image-tag 12.9.1-devel-ubi8 (CUDA 12)
```

Note the guard rejects a deliberately mismatched `--cuda-version`, which
previously ran (incorrectly). cc @aditranadive as this touches #1971 and
#2110.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Improved CUDA version detection for manylinux builds.
* Builds now derive the CUDA version from the selected base image when
not explicitly provided.
* Added validation to catch mismatched CUDA major versions and provide a
reliable fallback version.

* **Documentation**
  * Updated build help text to explain CUDA version derivation.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: NirWolfer <nwolfer@nvidia.com>
N
NirWolfer committed
538dba3682ba821acacf4630e4eed5aa42ba62d1
Parent: 6bff065
Committed by GitHub <noreply@github.com> on 8/20/2026, 5:43:56 PM