ci: merge L2 matrix entries so the e2e stage fits max-parallel: 10 (#3695)
* ci: merge L2 matrix entries so the e2e stage fits max-parallel: 10
cicd-e2e-tests declares max-parallel: 10 but carries 13 matrix entries, and
cicd-e2e-tests-gb200 carries 12. Every entry shares one runner label, so the
entries that do not fit the first wave cannot start until a wave-1 job frees a
slot. Measured over 4,175 CICD NeMo runs (2026-04-27..2026-08-26), that second
wave is now 43% of the L2 stage: the slowest suite's run time is flat at
13.8 -> 17.0 min while the wait for a runner went 0.2 -> 27.1 min.
Fold the cheapest suites into existing entries so each matrix fits its slot
budget. test-folder already accepts a comma-separated list and run_test.sh
splits on it, so no test is dropped, added, or reordered.
x86 13 -> 10: hf_transformer + hf_transformer_llm + data (14.8 min),
moe + parallelism -> L2_Distributed (8.6 min)
gb200 12 -> 9: the same two merges (10.5 and 6.2 min)
Every merged group stays under the stage floor set by the longest single suite
(L2_HF_Transformer_Finetune, 16.6 min x86; L2_Context_Parallel, 13.2 min gb200),
so the merge costs no wall-clock and removes the second wave entirely:
L2 stage 31.4 -> ~20.0 min median.
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
* ci: bound merged L2 job timeouts
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
---------
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> A
Alexandros Koumparoulis committed
cc6d64c31994af83217245aaf0d6e329c0cbc911
Parent: c1730fe
Committed by GitHub <noreply@github.com>
on 8/27/2026, 12:45:17 AM