SIGN IN SIGN UP

fix(irodori-tts): match upstream text normalization exactly

normalize_text had drifted from Irodori-TTS/irodori_tts/text_normalization.py
in ways that changed what the model is actually asked to read. All of these
predate v4 and affect every version:

- Sentence-final 。 and 、 were stripped. Upstream keeps them, and they carry
  prosody: dropping the 。 shortened predicted durations and clipped endings.
- ASCII spaces (U+0020) were removed, so "hello world" became "helloworld".
  Only the ideographic space U+3000 should go.
- ".." / "..." were not folded to "…".
- NFKC normalization was missing; only fullwidth alnum and halfwidth kana were
  approximated by hand, leaving ㈱ or Ⅲ untouched.
- Outer-bracket stripping ran once and ignored nesting depth, so
  「前半」と「後半」 lost its outer quotes and ((x)) only lost one level.

The caption is now stripped (not normalized), matching upstream, and the
reported token count uses the mask instead of the padded width.

Adds a differential test that runs both implementations over a fuzz corpus
when an upstream checkout is present, plus explicit cases for each behaviour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Y
yoshphys committed
0e846d23616079653b6fca89f4312837c5d94bc4
Parent: a9754f4