Hello,
I am trying to get feedback / ideas (specially from the people in --to) before I reach out to LKML.
Problem statement:
Share memory (dma-buf/dma-heap fd) over VSock (over virtio-msg-ffa).
Current status:
I have implemented the first version [1]/[2]. The system-heap memory is successfully shared with OPTEE, test setup available here [3].
Workflow:
Userspace:
open("/dev/dma_heap/system", O_RDWR) + ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc) + sendmsg(sock, &msg, 0) // msg: SCM_VSOCK_SHMEM / VSOCK_SHMEM_TYPE_FFA / VSOCK_SHMEM_SUBOP_OFFER
Kernel:
vsock_sendmsg_shmem(vsk, msg) | vsk->transport->map_dma_buf(dmabuf) | virtio_transport_map_dma_buf(dmabuf) | dma_buf_attach(dmabuf, dev); // Vsock's underlying device - virtio-msg device + dma_buf_map_attachment_unlocked(dbuf->attach, DMA_BIDIRECTIONAL); | system_heap_map_dma_buf(attach, dir); | dma_map_sgtable(attachment->dev, table, dir, attrs=0); | virtio_msg_dma_map_sg_dev(dev, sgl, nents, dir, attrs); // FFA dma-ops | ffa_dev->ops->mem_ops->memory_share(&args); + vmfdev->send(vmfdev, request, response, &vmfdev->idata);
The memory is shared with FFA SHARE type. This doesn't require a custom heap and works with any DMA heap/buf, system heap is just an example here.
FFA supports LEND-ing the memory as well and Google needs it. And this is where I am _blocked_ right now and need help.
We need to support both SHARE (both sides can access memory) and LEND (Linux doesn't access it any further - unmapped) mechanisms and I am not sure how to move forward.
- We can pass the share-type from userspace and that can reach up to vsock / dma-buf (with minor API change to dma_buf_map_attachment_unlocked()) / system-heap.
- But after that the standard dma_map_sgtable() call takes only one configurable parameter, `attributes` (which is set to 0 for now by system heap). There is nothing else which I can use to pass the flag to FFA specific DMA OPs.
- I thought maybe I can create a new DMA attribute: `DMA_ATTR_MEM_LEND`, which can be passed in the `attr` field and that will work I guess.
- But I am not sure if it would be acceptable upstream to add a new attribute for this.
- And so I am looking for some ideas.
On Wed, 28 Jan 2026 at 12:49, Viresh Kumar viresh.kumar@linaro.org wrote
We can pass the share-type from userspace and that can reach up to vsock / dma-buf (with minor API change to dma_buf_map_attachment_unlocked()) / system-heap.
But after that the standard dma_map_sgtable() call takes only one configurable parameter, `attributes` (which is set to 0 for now by system heap). There is nothing else which I can use to pass the flag to FFA specific DMA OPs.
I thought maybe I can create a new DMA attribute: `DMA_ATTR_MEM_LEND`, which can be passed in the `attr` field and that will work I guess.
But I am not sure if it would be acceptable upstream to add a new attribute for this.
If I understand you correctly, I think you need a dmabuf object that is mapped neither into the kernel nor into userspace, and to me that sounds like you should not be using the system heap for that, since that remains mapped into the kernel and is accessible from userspace as well.
The dma_map_sgtable() call in turn does not have to know about LEND vs SHARE, as all it does is map the memory into a bus address that can be shared with the firmware.
Arnd
On Wed, 28 Jan 2026 at 19:59, Arnd Bergmann arnd@linaro.org wrote:
If I understand you correctly, I think you need a dmabuf object that is mapped neither into the kernel nor into userspace, and to me that sounds like you should not be using the system heap for that, since that remains mapped into the kernel and is accessible from userspace as well.
That is doable with the help of a custom heap. Not an issue.
The dma_map_sgtable() call in turn does not have to know about LEND vs SHARE, as all it does is map the memory into a bus address that can be shared with the firmware.
The FFA memory share interface has a different interface for LEND, and so I need to distinguish LEND vs SHARE there as well in device specific dma-ops.
-- Viresh
+Arve Hjønnevåg arve@google.com
Viresh, as a first step we could follow the approach used in the Trusty driver with a vendor-specific secure heap which is FFA-aware and can select LEND on specific buffers as well as setting the FFA tag.
On Wed, Jan 28, 2026 at 7:01 AM Viresh Kumar viresh.kumar@linaro.org wrote:
On Wed, 28 Jan 2026 at 19:59, Arnd Bergmann arnd@linaro.org wrote:
If I understand you correctly, I think you need a dmabuf object that is mapped neither into the kernel nor into userspace, and to me that sounds like you should not be using the system heap for that, since that remains mapped into the kernel and is accessible from userspace as well.
That is doable with the help of a custom heap. Not an issue.
The dma_map_sgtable() call in turn does not have to know about LEND vs SHARE, as all it does is map the memory into a bus address that can be shared with the firmware.
The FFA memory share interface has a different interface for LEND, and so I need to distinguish LEND vs SHARE there as well in device specific dma-ops.
-- Viresh
On 28-01-26, 07:41, Armelle Laine wrote:
Viresh, as a first step we could follow the approach used in the Trusty driver with a vendor-specific secure heap which is FFA-aware and can select LEND on specific buffers as well as setting the FFA tag.
If the heap can be FFA aware, it can make direct FFA calls and doesn't need to use dma-ops of the virtio-msg device (Vsock). This is exactly what I did in the very first implementation. This will work.
Bertrand, rightly, objected to it and suggested to use the dma-ops instead so we can have a unified path to making FFA share operations.
So until we get a proper solution to this, we can do: - SHARE via system heap (with my current solution) - LEND via FFA aware heap (won't upstream)
Lets see if someone can suggest something else in the meantime.
On Wed, Jan 28, 2026 at 9:46 PM Viresh Kumar viresh.kumar@linaro.org wrote:
On 28-01-26, 07:41, Armelle Laine wrote:
Viresh, as a first step we could follow the approach used in the Trusty driver with a vendor-specific secure heap which is FFA-aware and can
select
LEND on specific buffers as well as setting the FFA tag.
If the heap can be FFA aware, it can make direct FFA calls and doesn't need to use dma-ops of the virtio-msg device (Vsock). This is exactly what I did in the very first implementation. This will work.
Bertrand, rightly, objected to it and suggested to use the dma-ops instead so we can have a unified path to making FFA share operations.
So until we get a proper solution to this, we can do:
- SHARE via system heap (with my current solution)
- LEND via FFA aware heap (won't upstream)
We also need to support "SEND_SECURE" when the FFA handle already exists. Shouldn't the heap rather be virtio-msg-ffa aware? wouldn't the heap also be responsible to invoke "FFA_BUS_MSG_AREA_SHARE" with both the FFA handle and the tag (for LEND or SEND_SECURE cases)?
Lets see if someone can suggest something else in the meantime.
-- viresh
On 28-01-26, 22:50, Armelle Laine wrote:
We also need to support "SEND_SECURE" when the FFA handle already exists.
Yeah, with our own heap this shouldn't be a problem.
Shouldn't the heap rather be virtio-msg-ffa aware? wouldn't the heap also be responsible to invoke "FFA_BUS_MSG_AREA_SHARE" with both the FFA handle and the tag (for LEND or SEND_SECURE cases)?
Right, that's what I implemented earlier. Virtio-msg-ffa aware.
Hi,
Just a general question on the subject: Do we want to unmap the area from user space or should we never map it ? I am mentioning this because side effects could create issues if something is mapped in user space application but in fact not accessible or removed (barriers required, be careful with possible side effects of speculation, cache flush required etc...). Might be a better solution if the application is just getting an address but can never access it ?
On 29 Jan 2026, at 08:18, Viresh Kumar viresh.kumar@linaro.org wrote:
On 28-01-26, 22:50, Armelle Laine wrote:
We also need to support "SEND_SECURE" when the FFA handle already exists.
Yeah, with our own heap this shouldn't be a problem.
Shouldn't the heap rather be virtio-msg-ffa aware? wouldn't the heap also be responsible to invoke "FFA_BUS_MSG_AREA_SHARE" with both the FFA handle and the tag (for LEND or SEND_SECURE cases)?
Right, that's what I implemented earlier. Virtio-msg-ffa aware.
Yes having a dedicated heap for that sounds like a better solution as passing the information through and handling the unmapping would require a new concept. But as Achin said maybe that is not a need just for our use case but more general in the future so we might want to investigate a generic lend solution.
Cheers Bertrand
-- viresh
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On 29-01-26, 07:29, Bertrand Marquis wrote:
Just a general question on the subject: Do we want to unmap the area from user space or should we never map it ?
I think maybe both, depending on the use case. But Google folks can confirm that the best as they understand the use cases well.
I am mentioning this because side effects could create issues if something is mapped in user space application but in fact not accessible or removed (barriers required, be careful with possible side effects of speculation, cache flush required etc...). Might be a better solution if the application is just getting an address but can never access it ?
The dma-buf API does provide some helpers to manage all that, I haven't used them yet but there is support to manage all this I think.
Yes having a dedicated heap for that sounds like a better solution as passing the information through and handling the unmapping would require a new concept. But as Achin said maybe that is not a need just for our use case but more general in the future so we might want to investigate a generic lend solution.
Yeah, maybe a generic LEND support is what we need.
Hi Viresh,
Some noob thoughts...
The problem of lending memory is more generic than FF-A. E.g. In Arm CCA memory is lent to the RMM by the Host. FF-A is just a mechanism to convey lent memory to the Borrower.
Is the key issue not that we need to enlighten dma-buf about the concept of lent memory/secure heap/protected memory? Could this not be solved by implementing a generic secure system heap as a "lend" equivalent of the existing system heap that is used for sharing memory. I believe this idea has been considered and vendor specific heaps are an intermediate step [1]?
Cheers, Achin
[1] https://source.android.com/docs/core/architecture/kernel/dma-buf-heaps
[cid:98a56555-6f6b-42c7-8161-32fb56dae1cd]https://outlook.office.com/bookwithme/user/caa0397acc4949e382b451677c9dc0c2@arm.com?anonymous&ismsaljsauthenabled&ep=owaSlotsEmailSignature Book time to meet with mehttps://outlook.office.com/bookwithme/user/caa0397acc4949e382b451677c9dc0c2@arm.com?anonymous&ismsaljsauthenabled&ep=owaSlotsEmailSignature ________________________________ From: Armelle Laine armellel@google.com Sent: 29 January 2026 06:50 To: Viresh Kumar viresh.kumar@linaro.org Cc: Arnd Bergmann arnd@linaro.org; Bertrand Marquis Bertrand.Marquis@arm.com; Achin Gupta Achin.Gupta@arm.com; Gil Cukierman cukie@google.com; Orlando Arbildo oarbildo@google.com; Ayrton Munoz (xWF) ayrton@google.com; Andrei Homescu (xWF) ahomescu@google.com; Ayrton Munoz (xWF) ayrton@xwf.google.com; Arve Hjønnevåg arve@google.com; virtio-msg@lists.linaro.org virtio-msg@lists.linaro.org; Vincent Guittot vincent.guittot@linaro.org; Lina Iyer lina.iyer@linaro.org; Sumit Semwal sumit.semwal@linaro.org Subject: Re: Share vs Lend - Vsock memory sharing over virtio-msg-ffa
On Wed, Jan 28, 2026 at 9:46 PM Viresh Kumar <viresh.kumar@linaro.orgmailto:viresh.kumar@linaro.org> wrote: On 28-01-26, 07:41, Armelle Laine wrote:
Viresh, as a first step we could follow the approach used in the Trusty driver with a vendor-specific secure heap which is FFA-aware and can select LEND on specific buffers as well as setting the FFA tag.
If the heap can be FFA aware, it can make direct FFA calls and doesn't need to use dma-ops of the virtio-msg device (Vsock). This is exactly what I did in the very first implementation. This will work.
Bertrand, rightly, objected to it and suggested to use the dma-ops instead so we can have a unified path to making FFA share operations.
So until we get a proper solution to this, we can do: - SHARE via system heap (with my current solution) - LEND via FFA aware heap (won't upstream)
We also need to support "SEND_SECURE" when the FFA handle already exists. Shouldn't the heap rather be virtio-msg-ffa aware? wouldn't the heap also be responsible to invoke "FFA_BUS_MSG_AREA_SHARE" with both the FFA handle and the tag (for LEND or SEND_SECURE cases)?
Lets see if someone can suggest something else in the meantime.
-- viresh IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On 29-01-26, 07:24, Achin Gupta wrote:
The problem of lending memory is more generic than FF-A. E.g. In Arm CCA memory is lent to the RMM by the Host. FF-A is just a mechanism to convey lent memory to the Borrower.
Absolutely.
Is the key issue not that we need to enlighten dma-buf about the concept of lent memory/secure heap/protected memory?
This is the first level problem, yes, and some part of it can be solved by enhancing the dma-buf API, for example by adding something like:
struct dma_buf_attachment * dma_buf_attach_with_flags(struct dma_buf *dmabuf, struct device *dev, u32 flags);
where the caller (Vsock) can send LEND/SHARE/etc in flags.
The problem I am stuck with is how to pass this special flag (behavior) via dma-ops to the core code making the change, i.e. virtio-msg FFA in this case.
Could this not be solved by implementing a generic secure system heap as a "lend" equivalent of the existing system heap that is used for sharing memory. I believe this idea has been considered and vendor specific heaps are an intermediate step [1]?
Hi Viresh,
On 29 Jan 2026, at 08:43, Viresh Kumar viresh.kumar@linaro.org wrote:
On 29-01-26, 07:24, Achin Gupta wrote:
The problem of lending memory is more generic than FF-A. E.g. In Arm CCA memory is lent to the RMM by the Host. FF-A is just a mechanism to convey lent memory to the Borrower.
Absolutely.
Is the key issue not that we need to enlighten dma-buf about the concept of lent memory/secure heap/protected memory?
This is the first level problem, yes, and some part of it can be solved by enhancing the dma-buf API, for example by adding something like:
struct dma_buf_attachment * dma_buf_attach_with_flags(struct dma_buf *dmabuf, struct device *dev, u32 flags);
where the caller (Vsock) can send LEND/SHARE/etc in flags.
The problem I am stuck with is how to pass this special flag (behavior) via dma-ops to the core code making the change, i.e. virtio-msg FFA in this case.
Shouldn't we add a new op to dma_ops so that it would also be easier to detect if a dma engine supports or not LEND ?
This would keep the current op fully compatible and have a clear way to identify and reject lend request on engine not supporting it.
Bertrand
Could this not be solved by implementing a generic secure system heap as a "lend" equivalent of the existing system heap that is used for sharing memory. I believe this idea has been considered and vendor specific heaps are an intermediate step [1]?
-- viresh
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
The implicit idea here that the vsock layer would be in control of the sharing type (SHARE vs LEND) is not correct. We typically don't expect the user space to be in control of this, but instead be known and enforced by the heaps. Achin, wouldn't his be also true for the host to RMM case?
On Wed, Jan 28, 2026 at 11:47 PM Bertrand Marquis Bertrand.Marquis@arm.com wrote:
Hi Viresh,
On 29 Jan 2026, at 08:43, Viresh Kumar viresh.kumar@linaro.org wrote:
On 29-01-26, 07:24, Achin Gupta wrote:
The problem of lending memory is more generic than FF-A. E.g. In Arm CCA memory is lent to the RMM by the Host. FF-A is just a mechanism to convey lent memory to the Borrower.
Absolutely.
Is the key issue not that we need to enlighten dma-buf about the concept of lent memory/secure heap/protected memory?
This is the first level problem, yes, and some part of it can be solved by enhancing the dma-buf API, for example by adding something like:
struct dma_buf_attachment * dma_buf_attach_with_flags(struct dma_buf *dmabuf, struct device *dev,
u32 flags);
where the caller (Vsock) can send LEND/SHARE/etc in flags.
The problem I am stuck with is how to pass this special flag (behavior) via dma-ops to the core code making the change, i.e. virtio-msg FFA in this case.
Shouldn't we add a new op to dma_ops so that it would also be easier to detect if a dma engine supports or not LEND ?
This would keep the current op fully compatible and have a clear way to identify and reject lend request on engine not supporting it.
Bertrand
Could this not be solved by implementing a generic secure system heap as a "lend" equivalent of the existing system heap that is used for sharing memory. I believe this idea has been considered and vendor specific heaps are an intermediate step [1]?
-- viresh
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On 28-01-26, 23:57, Armelle Laine wrote:
The implicit idea here that the vsock layer would be in control of the sharing type (SHARE vs LEND) is not correct. We typically don't expect the user space to be in control of this, but instead be known and enforced by the heaps.
Interesting, I was assuming until now that this information will come from userspace and that's what I wrote at multiple places earlier too. Thanks for pointing this out.
In that case I think we need a different heap for LEND/etc, SHARE can be done with system heap. The special heap may end up not using dma-ops at all but direct invocation of virtio-msg-ffa helpers.
On 29-01-26, 07:46, Bertrand Marquis wrote:
Shouldn't we add a new op to dma_ops so that it would also be easier to detect if a dma engine supports or not LEND ?
Maybe, I am not even sure if SHARE/LEND is something that we can describe as a property of the dma-ops and hence attributes. And so the whole confusion :(
-
Achin Gupta -
Armelle Laine -
Arnd Bergmann -
Bertrand Marquis -
Viresh Kumar