SIGN IN SIGN UP

[Bugfix][TE] Restore fork state on memory unregister to prevent VMA exhaustion (#3639) (#3660)

* [Bugfix][TE] Restore fork state on memory unregister to prevent VMA exhaustion (#3639)

ibv_reg_mr unconditionally calls madvise(MADV_DONTFORK) on registered
ranges when ibv_fork_init() has been called (which RdmaContext does at
startup).  The corresponding unregister path never issued MADV_DOFORK,
so VMAs remained permanently split even after the MR was destroyed.
Under high register/unregister churn with varying sizes (e.g. per-request
buffer registration in multimodal serving), this exhausts
vm.max_map_count and causes all subsequent ibv_reg_mr / ibv_create_qp
calls to fail with ENOMEM.

Fix: call madvise(addr, len, MADV_DOFORK) after successful ibv_dereg_mr
in both the rdma_transport and TENT unregister paths.  Failure to
restore is logged as a warning (non-fatal) — the deregister itself
already succeeded.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(rdma): cache addr/length before ibv_dereg_mr to avoid UAF

entry is freed by ibv_dereg_mr, so reading entry->addr/entry->length in
the error log and the subsequent MADV_DOFORK madvise was a use-after-free.
Cache region_addr/region_length before the dereg call. Also note the
deregister already succeeded when the madvise fails.

Addresses alogfans review on #3660.

* fix(rdma): cache addr/length before ibv_dereg_mr in unregisterMemoryRegion

The MR is freed by ibv_dereg_mr, so reading iter->second.mr->length in
the subsequent MADV_DOFORK madvise was a use-after-free. Cache region_addr
and region_length before the dereg call.

Addresses alogfans review on #3660.

* ci: re-trigger CI

* style(rdma): apply clang-format to unregisterMemReg fix (#3639)

* ci: re-trigger CI

* ci: re-trigger

* ci: re-trigger CI

---------

Co-authored-by: zhaoye <yzhao04@iquestlab.com>
Co-authored-by: Claude <noreply@anthropic.com>
H
Haoran FENG committed
489952835ca16b2ec59ec5c54c864e6248d5b562
Parent: b46235d
Committed by GitHub <noreply@github.com> on 8/26/2026, 2:48:18 AM