[TE] Add rdma_twosided control-plane notify channel (#3440)
* [TE] Add rdma_twosided control-plane notify channel Install RdmaTwoSidedTransport as an opt-in alternative to classic rdma and route sendNotify over a per-peer CtrlChannel, with OOB fallback. Co-authored-by: Cursor <cursoragent@cursor.com> * [TE] Harden twosided handshake decode and skip phantom RDMA NICs Reject out-of-range notify_rq_depth, avoid spinning on poll errors, and self-skip rdma_notify_test when CI enumerates an unusable mlx5_0. Co-authored-by: Cursor <cursoragent@cursor.com> * [TE] Serialize CtrlChannel connect and fix disconnect lock order Placeholder entries stop concurrent active handshakes to the same peer, and disconnect no longer nests send/resource mutexes in reverse order. Co-authored-by: Cursor <cursoragent@cursor.com> * [TE] Bound CtrlChannel connect waits and settle notify delivery on WC Address review feedback on #3440. ensureCtrlChannel (indefinite wait): - Distinguish an in-flight connect from a dead channel with a refcounted ctrl_connecting_ marker (refcounted because an active and a passive connect to the same peer can overlap). A waiter only parks when a connect is genuinely in flight; an entry that is !connected() and not in ctrl_connecting_ is dead and the next caller reclaims it. - Bound every wait with an absolute deadline (wait_until), configurable via MC_RDMA_NOTIFY_CONNECT_TIMEOUT_MS (default 10s); on expiry return nullptr and let the caller fall back to OOB notify. - ConnectScope (RAII) retracts the placeholder and notify_all()s on every exit path; the blocking handshake is wrapped in an UnlockGuard so the lock is guaranteed held at destruction. stopCtrlWorker() sets ctrl_stopping_ and wakes waiters during shutdown. CtrlChannel send (success concluded at WR submit): - Document fire-and-forget semantics: sendCtrlFrame/sendNotify return 0 == queued, not delivered; delivery is confirmed by the SEND WC. - Settle send accounting on the WC: an error WC clears the in-flight count (previously leaked) and drops connected_, so a delivery failure surfaces via the channel going unconnected + OOB fallback instead of a stale success. tests: add bounded-wait regression coverage (peer-gone, unreachable peer, concurrent unreachable). --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Z
zhtshr committed
3224c5b54850000bef145806d6d83d3b4305fc32
Parent: 845c685
Committed by GitHub <noreply@github.com>
on 8/24/2026, 2:26:20 AM