SIGN IN SIGN UP

Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.

0 0 104 Python

Fix EADDRINUSE errors in distributed tests with port manager and retry logic (#21309)

* Fix: Eliminate port collision race conditions in distributed tests

Implemented a thread-safe port reservation system to prevent EADDRINUSE
errors in distributed training tests.

- Created PortManager class with mutex-protected port allocation
- Updated find_free_network_port() to use PortManager
- Enhanced test teardown to release ports after completion
- Added 24 comprehensive tests (17 unit + 7 integration)
- Added context manager for automatic port cleanup

Fixes port collision issues in:
- tests_fabric/strategies/test_ddp_integration.py::test_clip_gradients
- tests_pytorch/strategies/test_fsdp.py::test_checkpoint_multi_gpus

* Fabric: reserve externally provided MASTER_PORT values

* Add port manager diagnostics and improve retry handling

- Add logging when port queue utilization >80% to detect exhaustion
- Enhance RuntimeError with detailed diagnostics (queue utilization %)
- Add DistNetworkError type check in pytorch conftest for subprocess failures
- Add test coverage for high queue utilization warning
- Helps diagnose EADDRINUSE issues in CI distributed tests

* chlog

* Empty-Commit

* Ensure EADDRINUSE retries pick a fresh port

- Drop cached MASTER_PORT before rerun so LightningEnvironment re-allocates
  a new port instead of reusing the TIME_WAIT socket
- Extend backoff to 1.0s to give the OS time to close TCPStore sockets
- Prevents NCCL connect errors caused by retries hitting the same port

---------

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: jirka <jirka.borovec@seznam.cz>
L
littlebullGit committed
6a8d94380e5b5e2b116b7e1e7edd23c57f42082f
Parent: f58a176
Committed by GitHub <noreply@github.com> on 10/23/2025, 2:40:53 PM