Ignore a stale best checkpoint recorded in a resumed trainer state (#48319)
* Ignore a stale best checkpoint recorded in a resumed trainer state Resuming from a checkpoint whose trainer_state.json points best_model_checkpoint at a directory that no longer exists (the usual case when checkpoints were moved to another machine or the original run directory was removed) kept the dead path all run long. If the resumed run never set a new best, training finished and then crashed in the save_total_limit=1 cleanup when os.path.samefile hit the missing path, and rotate_checkpoints counted the ghost path as protected, keeping one checkpoint beyond the configured limit for the whole run. Validate the restored path once at state load: warn and clear it so every consumer treats the run as having no best checkpoint yet. Test Plan: python -m pytest tests/trainer/test_trainer_checkpointing.py -k "stale" -q --no-header -p no:cacheprovider fail-before (validation reverted): 1 failed, FileNotFoundError from os.path.samefile inside _finalize_training pass-after: 1 passed python -m pytest tests/trainer/test_trainer_checkpointing.py -q --no-header -p no:cacheprovider 45 passed, 11 skipped (staging tests needing hub env) * Retrigger CI after runner-side checkout failures * Retrigger CI after a runner-side container kill
V
Vaggelis Giannopoulos committed
36deb0b53ed0863f4b4dfdea23dcaec7f3df3701
Parent: e15ea9b
Committed by GitHub <noreply@github.com>
on 8/26/2026, 6:02:17 PM