[TransferEngine] Split EFA transfers that straddle BufferDesc chunk boundaries (#3505)
EfaTransport::registerLocalMemoryInternal() already splits a buffer larger than min(max_mr_size, per-NIC PTE budget) into several chunks and publishes one BufferDesc per chunk, but submission did not follow: submitTransferTask() posted exactly one slice per request and selectDevice() only accepts a buffer that contains the WHOLE request. Any read/write crossing a seam therefore failed with ERR_ADDRESS_NOT_REGISTERED on the source side, or "Cannot select device for dest_addr" from EfaContext::submitPostSend() on the target side -- which is why mooncake-store still has to split EFA segments in GetTransportRegistrationLimit(). Mirror the RDMA transport: cap each posted slice at the remaining bytes of both the source and the target BufferDesc and look up the lkey per sub-slice (the target rkey is already resolved per slice in submitPostSend()). Unlike RDMA this deliberately does not also cut at globalConfig().slice_size -- EFA posts one WR per slice and SRD handles multi-MB messages natively, so a request that stays inside one chunk on both sides still becomes exactly one slice, as before. Chunks whose keys are all zero (a NIC the chunk was not registered on under the disjoint partition) are skipped, so capping never turns one miss into several. Slices already queued for other contexts are marked failed before returning an error, instead of being leaked. Also stop returning InvalidArgument when the advised device of a request is out of range or inactive: fall back to per-slice device selection instead. Adds efa_large_mr_test with four cases mirroring rdma_large_mr_test (baseline past the first chunk, target straddling, source straddling, and a READ straddling on both sides), forcing the split at a small size via MC_MAX_MR_SIZE. Like rdma_large_mr_test it is built but not registered with add_test(), because it needs EFA hardware. Fixes #3501 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
王
王鹤男 committed
09678aadfd1d3ed9f85b4b7b90d0ad09ed3596b3
Parent: 250d7fe
Committed by GitHub <noreply@github.com>
on 8/18/2026, 12:43:04 PM