studio: settle GPU VRAM after killing llama-server before the next reload (#5693)
* studio: settle GPU VRAM after killing llama-server before the next reload The NVIDIA driver reclaims a dead process's CUDA allocations asynchronously after the kernel reaps the PID -- typically tens to hundreds of milliseconds. Sampling `_get_gpu_free_memory` in that window reads artificially low, which propagates into `_select_gpus` / `_fit_context_to_vram` and flips the layer-split toward `--fit on` with more CPU-offloaded layers than steady-state would have required. On a tight VRAM card the resulting mmap thrash + OOM matches the Apply-reload kill path that bare-shell launches with the same flags never hit (continues the lineage of #5161 / #5401 / #5427). Adds `LlamaCppBackend._wait_for_vram_settle`: bounded poll of `_get_gpu_free_memory` that returns as soon as two consecutive samples agree per-GPU within `max(256 MiB, 2% of larger sample)`, or `max_wait` (default 2 s) wall-clock elapses with probe time included in the bound. Records `_last_kill_monotonic` inside `_kill_process`'s `finally` block so the wait engages on both in-process `load_model -> _kill_process -> load` and the frontend chat-settings Apply path (`/unload` then `/load`). The call site runs OUTSIDE the broad `self._lock` so concurrent `/unload`, `/cancel`, `/status` are not blocked during the wait. Short-circuits at zero cost on cold start (no kill recorded), stale kill (older than 15 s, driver has already settled), CPU-only host (probe returns empty), and probe exceptions (nvidia-smi gone away). 11 new unit tests in `test_llama_cpp_wait_for_vram_settle.py` cover: cold-start zero cost, stale-kill skip, slow-probe deadline bound, GPU index-set change, per-GPU stability with one draining card, the 2 % adaptive tolerance, _kill_process timestamp recording on real kill vs no-op, and an `inspect.getsource` contract that pins the call site to outside the Phase 3 lock and uses `_last_kill_monotonic` so a future refactor can't silently regress any of these properties. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Michael Han <michaelhan2050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
D
Daniel Han committed
2e1d0e2f19667e027135de03062ba99a24650243
Parent: 7482685
Committed by GitHub <noreply@github.com>
on 5/22/2026, 12:50:39 PM