Fix `gpt_oss` runs on GPU (#48118)
* tweak the kernels dep * Skip gpt_oss kernels-dependent tests without kernels * OOM guards * Make the CPU-RAM probe container-aware and guard host RAM in gpt-oss `psutil.virtual_memory().total` reads the host `/proc/meminfo`, so inside a K8S pod it reports the whole node (~750 GiB) and says nothing about the cgroup limit the OOM killer enforces. The `CI_CPU_MEMORY_LIMIT_GB` patch in `conftest.py` only half-covers this: it deliberately caps psutil at `limit * num_accelerators` (a `device_map="auto"` planning budget), which is 120 GiB on a 2-GPU runner whose cgroup still kills at 60. - add `get_cgroup_memory_limit_bytes()` (cgroup v2 `memory.max`, v1 `memory.limit_in_bytes`, with their respective unlimited sentinels) - add `get_cpu_ram_total_gib()`, taking the min of the cgroup limit, psutil, and the raw `CI_CPU_MEMORY_LIMIT_GB` - rebuild `require_large_cpu_ram` on it, and accept `@require_large_cpu_ram(memory=N)` as well as the bare form, matching `require_torch_large_accelerator`. This also fixes its two existing users (gemma2, longcat_flash), which were subject to the same host-vs-pod bug. - guard `test_model_outputs_distributed` on host RAM too: `distributed_worker` loads with no `device_map` and moves the model afterwards, so the whole checkpoint transits host RAM first -- that is the allocation that got the CI container SIGKILLed. Also cite the exact weight footprints from `modeling_utils.get_total_byte_count` on a meta-device model (38.96 GiB for 20b, 217.61 GiB for 120b) so the budgets are traceable rather than guessed. * Address review: separate the measured CPU RAM limit from the CI budget Follow-up to @ydshieh's review on #48118. `get_cpu_ram_total_gib` took the min of three values, one of which was `CI_CPU_MEMORY_LIMIT_GB` read raw. That variable is a *per-accelerator* budget, so on a 2-accelerator A10 runner it reads 60 GiB where the runner really has 180 -- every guard on that runner would over-skip. Multiplying by the accelerator count (what `conftest.py` already does when patching psutil) gets to 120, still not 180, because the budget is an allocation policy for `device_map="auto"` rather than a measurement of the machine. So split the two apart: - prefer what can be measured -- the cgroup limit the OOM killer enforces, and the machine's physical RAM -- taking the smaller, since each is wrong on its own (no cgroup outside a container; physical RAM reports the whole node inside a pod) - fall back to the CI budget only when neither can answer, i.e. on an instance-sharing runner with no cgroup limit of its own - add `get_ci_cpu_memory_budget_gib` as the single source of truth for the per-accelerator arithmetic, and have `conftest.py` use it instead of its own inline copy - add `get_physical_cpu_ram_gib`, which reads past the `patch_psutil_cpu_memory` cap. Without this the fallback above is unreachable in CI: conftest patches psutil at session start, so "physical RAM" would report the planning budget back to us rather than the machine. Return `inf` rather than 0 when nothing can answer at all -- no cgroup, no psutil, no CI budget. That is an ordinary local setup, not a broken one, and 0 silently skipped every memory-guarded test. `require_large_cpu_ram` loses its `is_psutil_available` early-return, now redundant; this also closes a gap where a container *without* psutil ran the test blind instead of consulting its cgroup limit. Also note in the docstrings that the 80 GiB default is "more than our 60 GiB GPU runners" rather than an estimate of any model, and that ROCm reaches `get_accelerator_total_memory_gib` through the cuda branch (a HIP build reports `torch_device == "cuda"`), since the existing comment read as though it were excluded. Tests cover the resolution order in all four environments, including that the guard reads past the psutil cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
T
Tarek Ziade committed
275b5245f5e23e52d87332a6bce88fb639d47c4a
Parent: 861f4c4
Committed by GitHub <noreply@github.com>
on 8/21/2026, 1:35:12 PM