SIGN IN SIGN UP

int4 per-token-head KV cache with DFlash2: 256k pool on one card (#42)

Rebase of PR #42 onto this patch stack. The submitted patch was cut against
stock 0.27.1 and re-shipped hybrid-sw-block-promote.patch functions plus an
unreachable mamba GCD-hash branch; both dropped, the three real fixes kept:
the padded-page per-block narrow in attn_utils, the causal-only assert
removal, and per-seq-causal plumbing through the int4 Triton kernel.

Verified on the 3090: unpatched boot reproduces the RuntimeError shape
mismatch at engine init; patched serves 314,915 tokens of pool at 256k
max-model-len, tool calling round-trips, lookup lane active. Shipped
FLASH_ATTN bf16 config inert: pool byte-identical at 57,669.
alternative.sh parameterized to house conventions (env overrides, WSL
GPU_UTIL note, stale-shm cleanup, DFLASH_TOKENS>7 sync-scheduler rule).

Co-authored-by: Walid Lachhab <lachhabw@users.noreply.github.com>
M
mhenrichsen committed
04edd2bbf11d6398306581113c84f1f062f60190
Parent: 60daef8