SIGN IN SIGN UP

fix: normalize NLLanguage code so Chinese hits the no-space word split path (#511)

* fix: normalize NLLanguage code so Chinese hits the no-space word split path

NLLanguageRecognizer returns BCP-47 tags ("zh-Hans"/"zh-Hant") while the
no-space-language whitelist in splitToWordTokens uses Whisper-style codes
("zh"), so Chinese text never took the splitTokensOnUnicode path. Clauses
were glued into single "words" by the space-based splitter, then truncated
by the max-word-duration heuristic (min(0.7, median) * 2 = 1.4s), producing
severely wrong word timestamps for zh-Hans, zh-Hant, and Cantonese.

Normalize the detected language through Locale so both Chinese variants
map to "zh", matching openai/whisper which checks its decoding language
code directly. No behavior change for other languages.

Adds testSplitToWordTokensChinese mirroring the Japanese test; it fails
before this change and passes after.

* test(whisperkit): add Traditional Chinese (zh-Hant) word-token split test

Verify that Traditional Chinese correctly normalizes to 'zh' and hits
the no-space Unicode split path, same as Simplified Chinese (zh-Hans).
Includes both conservative guard assertions (path detection + token
preservation) and exact expected-split assertions.
F
Freecodetiger committed
d9dc4baa50d16773ecd5d1d7fea4b7c20c1ab79a
Parent: f5f989d
Committed by GitHub <noreply@github.com> on 7/28/2026, 10:02:07 PM