speculative : add heuristic algorithm (#3006)
* Add heuristic algo for speculative * Constrain minimum n_draft to 2 * speculative : improve heuristic impl * speculative : be more rewarding upon guessing max drafted tokens * speculative : fix typos --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
L
Leng Yue committed
35f73049af6c676a106a5a990a819ae0bc3fcd7d
Parent: 71ca2fa
Committed by GitHub <noreply@github.com>
on 9/14/2023, 4:14:44 PM