SIGN IN SIGN UP

feat(cli): suggest SSD-only caching on low-memory Macs at session startup

The in-RAM paged KV mirror (L1) shares unified memory with the model's
weights. On a 16-32GB Mac that trade is usually a loss, and the SSD block
cache (L2) gives nearly all of the benefit without taking memory the model
needs. Measured on Qwen3.8-27B at default capacities, same workload, only the
RAM mirror differing:

  paged RAM + L2   cold 20.18s   full hit 2.68s (7.5x)   partial 3.48s (5.8x)
  SSD-only         cold 19.81s   full hit 2.70s (7.3x)   partial 3.57s (5.5x)

Within about 2 percent. The suggestion therefore costs the user almost nothing
in speed while freeing gigabytes for weights.

It is a suggestion and nothing else. It does not disable the paged cache, does
not mutate any argument, and does not refuse to start. An invented RAM guard
once refused to load big models across six releases on exactly the hardware
this product exists for, so a guess about memory must never acquire veto power
over a user's configuration. When RAM cannot be detected the advisory stays
silent rather than becoming a warning of its own, because a failed detection
must not turn into a user-facing restriction either.

It only appears when the paged cache is actually on, calls out when the L2
cache is off so the advice is actionable, and the threshold is tunable through
VMLX_LOW_RAM_ADVISORY_GB for support cases. Tests cover the ladder from 16GB
to 128GB, silence on failed detection, and an assertion that the advisory
leaves the parsed arguments byte-identical.
J
Jinho Jang committed
200e6ba7ccd14872d510fe74f55df4c0a82dcc38
Parent: 8208b36