On Thu, 2 Jul 2026 00:56:40 -0400 David Hu xuehaohu@google.com wrote:
On Tue, Jun 30, 2026 at 8:42 AM Jason Gunthorpe jgg@ziepe.ca wrote:
On Tue, Jun 23, 2026 at 11:53:50PM +0100, David Laight wrote:
If we restrict incoming dmabuf transfers to fit within VFS-centric limits (2GB), we impose unnecessary overhead on the RDMA stack, forcing it to manage a significantly higher number of memory registrations. By cleanly splitting these massive contiguous device buffers into page-aligned SGL entries, we directly improve the efficiency of P2P transfers and memory registration.
But a divide by '4G - PAGE_SIZE' is also non-trivial and (I think affects a lot of io) when the quotient is always 1. Splitting into 2G chunks is a lot cheaper.
Doesn't matter this isn't fast path stuff. It is better to use fewer SGL entries, IHMO.
Since this change doesn't seem to have a negative impact on standard file I/O or break existing VFS constraints, I'm curious why we shouldn't support splitting these >4GB P2P transfers? Am I missing something?
I was only wondering whether it was needed... It does bring up the question of why the >4GB transfers even need splitting. But that is another question.
SGL can only store an unsigned int size, so any large physical range has to be split down.
rdma now a days has code to process the sgl and restore back the > 4G sizes since mode RDMA HW can accept that.
commit 486055f5e09df959ad4e3aa4ee75b5c91ddeec2e Author: Michael Margolin mrgolin@amazon.com Date: Mon Feb 17 14:16:23 2025 +0000
RDMA/core: Fix best page size finding when it can cross SG entriesSo whatever this produces needs to be compatible with that to undo it.
Thank you everyone. It looks like most open issues are sorted out. I'll wait for maintainers to weigh in before sending out v3 (which will remove the type cast for min() per David L.'s feedback, and revert to ALIGN_DOWN(UINT_MAX, PAGE_SIZE) per Jason's feedback).
Does this code get used a lot for 'normal' transfers? I'm away from my normal systems and can't check. But if pretty much all of the fragments are small (< 4G) then it is probably worth adding a check for 'size < limit' before anything else and optimising that case.
David
Hi Jason,
Thank you for your feedback. I took a closer look at the commit to ensure compatibility. This patch is perfectly complementary, and actually prevents a failure in an edge case for the latest `ib_umem_find_best_pgsz` [1].
Regards, David
[1] For dma-buf split with `0xFFFFFFFF`, in case of a discontinguity in later buffers, we will hit this code path in `ib_umem_find_best_pgsz`
if (i != 0) mask |= va;(*After `va` had been incremented by `0xFFFFFFFF`, due to `va += sg_dma_len(sg) - pgoff`) (*Which will set the lowest bit of `mask` to 1)
Because `count_trailing_zeros(mask) returns 0`, `ib_umem_find_best_pgsz()` will always return 0 in such cases.