SIGN IN SIGN UP

[TransferEngine] Split EFA transfers that straddle BufferDesc chunk boundaries (#3505)

EfaTransport::registerLocalMemoryInternal() already splits a buffer larger
than min(max_mr_size, per-NIC PTE budget) into several chunks and publishes
one BufferDesc per chunk, but submission did not follow: submitTransferTask()
posted exactly one slice per request and selectDevice() only accepts a buffer
that contains the WHOLE request. Any read/write crossing a seam therefore
failed with ERR_ADDRESS_NOT_REGISTERED on the source side, or "Cannot select
device for dest_addr" from EfaContext::submitPostSend() on the target side --
which is why mooncake-store still has to split EFA segments in
GetTransportRegistrationLimit().

Mirror the RDMA transport: cap each posted slice at the remaining bytes of
both the source and the target BufferDesc and look up the lkey per sub-slice
(the target rkey is already resolved per slice in submitPostSend()). Unlike
RDMA this deliberately does not also cut at globalConfig().slice_size -- EFA
posts one WR per slice and SRD handles multi-MB messages natively, so a
request that stays inside one chunk on both sides still becomes exactly one
slice, as before. Chunks whose keys are all zero (a NIC the chunk was not
registered on under the disjoint partition) are skipped, so capping never
turns one miss into several. Slices already queued for other contexts are
marked failed before returning an error, instead of being leaked.

Also stop returning InvalidArgument when the advised device of a request is
out of range or inactive: fall back to per-slice device selection instead.

Adds efa_large_mr_test with four cases mirroring rdma_large_mr_test
(baseline past the first chunk, target straddling, source straddling, and a
READ straddling on both sides), forcing the split at a small size via
MC_MAX_MR_SIZE. Like rdma_large_mr_test it is built but not registered with
add_test(), because it needs EFA hardware.

Fixes #3501

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
王鹤男 committed
09678aadfd1d3ed9f85b4b7b90d0ad09ed3596b3
Parent: 250d7fe
Committed by GitHub <noreply@github.com> on 8/18/2026, 12:43:04 PM