SIGN IN SIGN UP

feat(irodori-tts): add v4 support with shared pretrained text encoder

Irodori-TTS v4-Small replaces the two scratch-trained text/caption
encoders with a single pretrained ModernBERT-ja-310m backbone feeding
separate projectors, and raises speaker_patch_size to 4.

- Port ModernBERT to MLX (modernbert.py): fused Wqkv, GeGLU MLP,
  alternating global/sliding-window attention with per-layer-type RoPE
  theta, and layer 0's identity attn_norm. Verified against
  transformers on random weights to 5e-7.
- Add PretrainedTextBackbone + PretrainedConditionProjector and route
  text/caption encoding through them when text_encoder_type is
  "pretrained"; scratch encoders remain the default for v1-v3.
- Load the tokenizer bundled with the converted weights when present,
  so inference no longer depends on the upstream tokenizer repo.
- Support multi-clip reference audio: clips are encoded separately and
  concatenated, then trimmed to the checkpoint's ref_max_seconds (120s
  for v4, 30s before).
- Size the no-reference placeholder latent to speaker_patch_size, which
  otherwise patches down to an empty sequence at v4's patch size 4.

The sliding-window mask keeps its diagonal open so padding queries far
from any real token cannot produce an all-masked softmax row.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Y
yoshphys committed
98a021d05aaa2084f37c004b09e35f400475bc57
Parent: 41aba81