Add AWS EFA support to rdmaproxy
Add an efaproxy driver plug-in for AWS Elastic Fabric Adapter (EFA) devices
bound to the host `efa` driver. EFA's driver-private uverbs ABI carries no
userspace buffer pointers on CQ/QP CREATE: queue memory is kernel-allocated
and handed back via mmap keys in the CREATE response, which the core already
forwards to the host FD verbatim. PrepareCreateDMA is therefore a no-op; the
plug-in exists to register the "efa" driver name and to model EFA's MR_QUERY
driver-namespace method (used by libfabric to fetch RDMA read/write
interconnect IDs).
Model the standard AH object's destroy method (UVERBS_OBJECT_AH): EFA (SRD)
creates address handles via the legacy write path and destroys them through
the modern object, which the core must allow.
Extend the virtual RDMA sysfs for libfabric/hwloc topology discovery:
- device/driver symlink and /sys/bus/pci/drivers/<driver> tree, which
libfabric's EFA provider resolves during device discovery.
- Mirror each PCI node's raw config space ("config" file). hwloc (used by
aws-ofi-nccl to build the NCCL topology) reads it to recover the PCI
bridge bus-number registers; without it hwloc cannot reconstruct the
bridge hierarchy and aws-ofi-nccl aborts its topology write.
- Serve the per-port lid_mask_count attribute alongside the other port
attributes libibverbs reads from sysfs; ib_core creates it on every
port.
The sysfs integration tests cover the new driver tree and config mirroring.
Advertise a >= 5.12 kernel release in RDMA-enabled sandboxes so aws-ofi-nccl
enables dmabuf-based GPU memory registration (ibv_reg_dmabuf_mr), which the
proxy handles via the existing REG_DMABUF_MR path. Sandboxes without RDMA
keep the current 4.19 release to limit the bump's blast radius; a later
change can make the newer release the default.
Co-authored-by: Alessio Ricci Toniolo <alessio@modal.com> A
Ayush Ranjan committed
d20f077d1cffe039c7c4690e5c8ee63668053a41
Parent: 32d6f00