Optimize pure LoRA prefill routing (#647)
This PR is: - To optimize pure LoRA prefill by applying adapter deltas per active slot instead of expanding LoRA weights per token. - To reduce long-prompt prefill latency and temporary adapter-weight traffic. - To keep decode and mixed decode/prefill batches on the existing routing path. Before: each LoRA token gathered full A/B adapter matrices, so prefill copied the same adapter weights once per token. After: pure prefill groups rows by adapter slot and reuses each adapter A/B matrix for the whole row range or row group. Scope note: this intentionally optimizes pure prefill only. If any one-token decode row is present, the batch keeps the existing expanded route to avoid decode regression. Local hot-path microbenchmark, lower ms is better: | Tokens | Before old expanded route | After pure-prefill route | Speedup | |---:|---:|---:|---:| | 64 | 0.773 ms | 0.563 ms | 1.37x | | 512 | 1.006 ms | 0.260 ms | 3.87x | | 2048 | 1.175 ms | 0.406 ms | 2.90x | | 4096 | 2.490 ms | 0.449 ms | 5.54x | --------- Signed-off-by: Yuan Lik Xun <lxyuan0420@gmail.com>
L
Lik Xun Yuan (Lx) committed
14705ad974863f68d00315655514f200366441bf
Parent: 813e738
Committed by GitHub <noreply@github.com>
on 8/28/2026, 8:55:47 AM