Avoid unnecessarily disabling CUDA graphs (#7302)
As discussed in PR #6766, CUDA graphs were being disabled in the presence of long prompts. This fixes the issue by avoiding the consective update counter from incrementing unnecessarily for tokens in which cuda graphs are disabled due to batch size > 1.
A
agray3 committed
dc020985b8755dd6aa93a2f002f43c3ede808cce
Parent: 344f912
Committed by GitHub <noreply@github.com>
on 5/15/2024, 1:44:49 PM