Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.
Fix EADDRINUSE errors in distributed tests with port manager and retry logic (#21309)
* Fix: Eliminate port collision race conditions in distributed tests Implemented a thread-safe port reservation system to prevent EADDRINUSE errors in distributed training tests. - Created PortManager class with mutex-protected port allocation - Updated find_free_network_port() to use PortManager - Enhanced test teardown to release ports after completion - Added 24 comprehensive tests (17 unit + 7 integration) - Added context manager for automatic port cleanup Fixes port collision issues in: - tests_fabric/strategies/test_ddp_integration.py::test_clip_gradients - tests_pytorch/strategies/test_fsdp.py::test_checkpoint_multi_gpus * Fabric: reserve externally provided MASTER_PORT values * Add port manager diagnostics and improve retry handling - Add logging when port queue utilization >80% to detect exhaustion - Enhance RuntimeError with detailed diagnostics (queue utilization %) - Add DistNetworkError type check in pytorch conftest for subprocess failures - Add test coverage for high queue utilization warning - Helps diagnose EADDRINUSE issues in CI distributed tests * chlog * Empty-Commit * Ensure EADDRINUSE retries pick a fresh port - Drop cached MASTER_PORT before rerun so LightningEnvironment re-allocates a new port instead of reusing the TIME_WAIT socket - Extend backoff to 1.0s to give the OS time to close TCPStore sockets - Prevents NCCL connect errors caused by retries hitting the same port --------- Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> Co-authored-by: jirka <jirka.borovec@seznam.cz>
L
littlebullGit committed
6a8d94380e5b5e2b116b7e1e7edd23c57f42082f
Parent: f58a176
Committed by GitHub <noreply@github.com>
on 10/23/2025, 2:40:53 PM