Keep the hybrid APC parity harness bootable on 16 GB hosts (#639)
Fixes #638. ## Summary The `--quick` gate (and the defaults arm in general) passed no `max_num_batched_tokens`. `LLM()` therefore took the 8192 LLM-class default, and `profile_run()` evaluated a single `(1, 8192)` dummy. In the reproduced configuration on the tested 16 GB M-series Mac, the cache-on child's profiled overhead reached 6.46 GB, its KV budget reached -1.81 GB, and it died during `LLM()` initialization. The cache-off child on the same revision initialized with 2.38 GB of overhead. This change is limited to `tools/hybrid_apc_parity_matrix.py`: - The defaults arm now passes `max_num_batched_tokens=2048`, matching its existing `max_model_len`. The arm no longer exercises the LLM-class 8192 default. That default was not needed by the short parity prompts; it made the profile dummy four times longer than any legal sequence in this harness. The explicit `mnbt1088` arm is unchanged. - The parent polls the queue in five-second intervals and fails immediately with the child exit code if the child terminates before `queue.put()`, instead of waiting up to 1500 seconds and then raising `queue.Empty`. This PR changes no serving behavior or library default. A serving-shaped profile path is separate work being discussed in RFC #525 and #590. #585/#586 concerned the former MLX per-buffer default; #637's proposed conditional default does not apply below 50 GiB usable memory and is not involved on the tested host. ## Testing - `python tools/hybrid_apc_parity_matrix.py --quick` — 20 comparisons, 0 mismatches, `PARITY PASS` - Cache-on profile: overhead 6.46 GB -> 0.73 GB; KV budget -1.81 GB -> +3.92 GB - Cache-off bounded profile: overhead 0.76 GB; final KV budget +3.83 GB The end-to-end wrapper is marked `slow`, is not run in CI, and requires local Qwen/Qwen3.5-0.8B weights. ## AI assistance AI assistance was used in developing this change. The problem analysis, the patch, and the measurements were reviewed and verified by me, and I am responsible for all submitted changes. Signed-off-by: Polyphonia.it <noreply@polyphonia.it> Co-authored-by: Polyphonia.it <noreply@polyphonia.it>
F
fdaluiso committed
6ef8313ec94c821f7bea9be6bb3e7198fc899329
Parent: 41f3ed3
Committed by GitHub <noreply@github.com>
on 8/25/2026, 3:48:06 AM