[Bugfix][TE] Restore fork state on memory unregister to prevent VMA exhaustion (#3639) (#3660)
* [Bugfix][TE] Restore fork state on memory unregister to prevent VMA exhaustion (#3639) ibv_reg_mr unconditionally calls madvise(MADV_DONTFORK) on registered ranges when ibv_fork_init() has been called (which RdmaContext does at startup). The corresponding unregister path never issued MADV_DOFORK, so VMAs remained permanently split even after the MR was destroyed. Under high register/unregister churn with varying sizes (e.g. per-request buffer registration in multimodal serving), this exhausts vm.max_map_count and causes all subsequent ibv_reg_mr / ibv_create_qp calls to fail with ENOMEM. Fix: call madvise(addr, len, MADV_DOFORK) after successful ibv_dereg_mr in both the rdma_transport and TENT unregister paths. Failure to restore is logged as a warning (non-fatal) — the deregister itself already succeeded. Co-Authored-By: Claude <noreply@anthropic.com> * fix(rdma): cache addr/length before ibv_dereg_mr to avoid UAF entry is freed by ibv_dereg_mr, so reading entry->addr/entry->length in the error log and the subsequent MADV_DOFORK madvise was a use-after-free. Cache region_addr/region_length before the dereg call. Also note the deregister already succeeded when the madvise fails. Addresses alogfans review on #3660. * fix(rdma): cache addr/length before ibv_dereg_mr in unregisterMemoryRegion The MR is freed by ibv_dereg_mr, so reading iter->second.mr->length in the subsequent MADV_DOFORK madvise was a use-after-free. Cache region_addr and region_length before the dereg call. Addresses alogfans review on #3660. * ci: re-trigger CI * style(rdma): apply clang-format to unregisterMemReg fix (#3639) * ci: re-trigger CI * ci: re-trigger * ci: re-trigger CI --------- Co-authored-by: zhaoye <yzhao04@iquestlab.com> Co-authored-by: Claude <noreply@anthropic.com>
H
Haoran FENG committed
489952835ca16b2ec59ec5c54c864e6248d5b562
Parent: b46235d
Committed by GitHub <noreply@github.com>
on 8/26/2026, 2:48:18 AM