fix(staging): source tokenizer from the model_dir; ship Flux2Klein bundle
The Model Auto Loader (Klein-9B + a separately-selected transformer) failed at pipeline build with "Failed to open vocab.json: <staging>/tokenizer/vocab.json". Staging called copy_tokenizer_bundle(arch=Flux2Klein), which only looks in <plugin>/bin/tokenizers/<arch>/ — a dir that is not shipped to users — and ignored the complete tokenizer/ that ships inside the already-downloaded base model. Fix: - hf_layout.copy_tokenizer(arch, dst, candidate_model_dirs): prefer a tokenizer/ found in a source model_dir (HF base-model / exported-separated downloads ship one), fall back to the plugin bundle. Zero extra download, zero user action. - hf_native.adapt: the fresh-staging path passes the source model_dirs to copy_tokenizer; the existing-model_dir early-return now backfills a missing tokenizer from the bundle so an export without tokenizer/ is still loadable. - Ship bin/tokenizers/Flux2Klein/ (vocab/merges/tokenizer json + configs) so the single-file modes that have no model_dir to borrow from — bundled all-in-one checkpoint, qf_flat export, ComfyUI UNETLoader — get a tokenizer too. (.gitignore already tracks non-binary bin files.) - factory._STAGING_FORMAT_VERSION=2: invalidate cached staging dirs from before this fix (they carry a `.staging_complete` marker but an empty tokenizer/), so users only need to pull + restart — no manual cleanup. Validated: 14/14 Linux unit tests + 8/8 on a real Windows Server 2022 / NTFS box (model_dir-preferred copy, bundle fallback, early-return backfill, cross-platform path derivation, cache-version invalidation). Plain file copy — no symlink, so no Windows privilege dependency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Y
yejianhui committed
130ef4329bcbce3596dcb83e0f471af41522c9b1
Parent: 35477a6