SIGN IN SIGN UP

fix(staging): source tokenizer from the model_dir; ship Flux2Klein bundle

The Model Auto Loader (Klein-9B + a separately-selected transformer) failed at
pipeline build with "Failed to open vocab.json: <staging>/tokenizer/vocab.json".
Staging called copy_tokenizer_bundle(arch=Flux2Klein), which only looks in
<plugin>/bin/tokenizers/<arch>/ — a dir that is not shipped to users — and
ignored the complete tokenizer/ that ships inside the already-downloaded base
model.

Fix:
- hf_layout.copy_tokenizer(arch, dst, candidate_model_dirs): prefer a tokenizer/
  found in a source model_dir (HF base-model / exported-separated downloads ship
  one), fall back to the plugin bundle. Zero extra download, zero user action.
- hf_native.adapt: the fresh-staging path passes the source model_dirs to
  copy_tokenizer; the existing-model_dir early-return now backfills a missing
  tokenizer from the bundle so an export without tokenizer/ is still loadable.
- Ship bin/tokenizers/Flux2Klein/ (vocab/merges/tokenizer json + configs) so the
  single-file modes that have no model_dir to borrow from — bundled all-in-one
  checkpoint, qf_flat export, ComfyUI UNETLoader — get a tokenizer too.
  (.gitignore already tracks non-binary bin files.)
- factory._STAGING_FORMAT_VERSION=2: invalidate cached staging dirs from before
  this fix (they carry a `.staging_complete` marker but an empty tokenizer/), so
  users only need to pull + restart — no manual cleanup.

Validated: 14/14 Linux unit tests + 8/8 on a real Windows Server 2022 / NTFS box
(model_dir-preferred copy, bundle fallback, early-return backfill, cross-platform
path derivation, cache-version invalidation). Plain file copy — no symlink, so no
Windows privilege dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Y
yejianhui committed
130ef4329bcbce3596dcb83e0f471af41522c9b1
Parent: 35477a6