test(checkpoint): redesign resume robustness around a shared trajectory (#3427)

* test(checkpoint): use shared trajectory for resume checks

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* test(checkpoint): report independent training reproducibility

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* test(checkpoint): surface reproducibility threshold alerts

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* fix(checkpoint): source reproducibility batches from config

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* fix(test): verify restored dataloader by resumed batches

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* test(ci): calibrate checkpoint robustness envelopes

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

* test(ci): relax retriever resume loss envelope

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

---------

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>