int4 per-token-head KV cache with DFlash2: 256k pool on one card (#42)
Rebase of PR #42 onto this patch stack. The submitted patch was cut against stock 0.27.1 and re-shipped hybrid-sw-block-promote.patch functions plus an unreachable mamba GCD-hash branch; both dropped, the three real fixes kept: the padded-page per-block narrow in attn_utils, the causal-only assert removal, and per-seq-causal plumbing through the int4 Triton kernel. Verified on the 3090: unpatched boot reproduces the RuntimeError shape mismatch at engine init; patched serves 314,915 tokens of pool at 256k max-model-len, tool calling round-trips, lookup lane active. Shipped FLASH_ATTN bf16 config inert: pool byte-identical at 57,669. alternative.sh parameterized to house conventions (env overrides, WSL GPU_UTIL note, stale-shm cleanup, DFLASH_TOKENS>7 sync-scheduler rule). Co-authored-by: Walid Lachhab <lachhabw@users.noreply.github.com>
M
mhenrichsen committed
04edd2bbf11d6398306581113c84f1f062f60190
Parent: 60daef8