Fix: Missing token count for GPT-4o model. (bumps tiktoken-rs to v0.5.9) (#11893)
Fix: this makes sure we have token counts for the new GPT-4o model. See: https://github.com/zurawiki/tiktoken-rs/releases/tag/v0.5.9 Release Notes: - Fix: Token count was missing for the new GPT-4o model. (I believe this should go in a 0.136.x release)
T
Toon Willems committed
9969d6c702c64823787e88ccd104a2068dcd6d81
Parent: 8c8c176
Committed by GitHub <noreply@github.com>
on 5/16/2024, 11:09:28 AM