SIGN IN SIGN UP

Optimize pure LoRA prefill routing (#647)

This PR is:

- To optimize pure LoRA prefill by applying adapter deltas per active
slot instead of expanding LoRA weights per token.
- To reduce long-prompt prefill latency and temporary adapter-weight
traffic.
- To keep decode and mixed decode/prefill batches on the existing
routing path.

Before: each LoRA token gathered full A/B adapter matrices, so prefill
copied the same adapter weights once per token.

After: pure prefill groups rows by adapter slot and reuses each adapter
A/B matrix for the whole row range or row group.

Scope note: this intentionally optimizes pure prefill only. If any
one-token decode row is present, the batch keeps the existing expanded
route to avoid decode regression.

Local hot-path microbenchmark, lower ms is better:

| Tokens | Before old expanded route | After pure-prefill route |
Speedup |
|---:|---:|---:|---:|
| 64 | 0.773 ms | 0.563 ms | 1.37x |
| 512 | 1.006 ms | 0.260 ms | 3.87x |
| 2048 | 1.175 ms | 0.406 ms | 2.90x |
| 4096 | 2.490 ms | 0.449 ms | 5.54x |

---------

Signed-off-by: Yuan Lik Xun <lxyuan0420@gmail.com>
L
Lik Xun Yuan (Lx) committed
14705ad974863f68d00315655514f200366441bf
Parent: 813e738
Committed by GitHub <noreply@github.com> on 8/28/2026, 8:55:47 AM