SIGN IN SIGN UP

[TE] Add rdma_twosided control-plane notify channel (#3440)

* [TE] Add rdma_twosided control-plane notify channel

Install RdmaTwoSidedTransport as an opt-in alternative to classic rdma
and route sendNotify over a per-peer CtrlChannel, with OOB fallback.

Co-authored-by: Cursor <cursoragent@cursor.com>

* [TE] Harden twosided handshake decode and skip phantom RDMA NICs

Reject out-of-range notify_rq_depth, avoid spinning on poll errors, and
self-skip rdma_notify_test when CI enumerates an unusable mlx5_0.

Co-authored-by: Cursor <cursoragent@cursor.com>

* [TE] Serialize CtrlChannel connect and fix disconnect lock order

Placeholder entries stop concurrent active handshakes to the same peer,
and disconnect no longer nests send/resource mutexes in reverse order.

Co-authored-by: Cursor <cursoragent@cursor.com>

* [TE] Bound CtrlChannel connect waits and settle notify delivery on WC

Address review feedback on #3440.

ensureCtrlChannel (indefinite wait):
- Distinguish an in-flight connect from a dead channel with a refcounted
  ctrl_connecting_ marker (refcounted because an active and a passive
  connect to the same peer can overlap). A waiter only parks when a connect
  is genuinely in flight; an entry that is !connected() and not in
  ctrl_connecting_ is dead and the next caller reclaims it.
- Bound every wait with an absolute deadline (wait_until), configurable via
  MC_RDMA_NOTIFY_CONNECT_TIMEOUT_MS (default 10s); on expiry return nullptr
  and let the caller fall back to OOB notify.
- ConnectScope (RAII) retracts the placeholder and notify_all()s on every
  exit path; the blocking handshake is wrapped in an UnlockGuard so the lock
  is guaranteed held at destruction. stopCtrlWorker() sets ctrl_stopping_
  and wakes waiters during shutdown.

CtrlChannel send (success concluded at WR submit):
- Document fire-and-forget semantics: sendCtrlFrame/sendNotify return 0 ==
  queued, not delivered; delivery is confirmed by the SEND WC.
- Settle send accounting on the WC: an error WC clears the in-flight count
  (previously leaked) and drops connected_, so a delivery failure surfaces
  via the channel going unconnected + OOB fallback instead of a stale
  success.

tests: add bounded-wait regression coverage (peer-gone, unreachable peer,
concurrent unreachable).

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Z
zhtshr committed
3224c5b54850000bef145806d6d83d3b4305fc32
Parent: 845c685
Committed by GitHub <noreply@github.com> on 8/24/2026, 2:26:20 AM