All,
We have had a number of spam emails on this list in the past few weeks.
I set all spam authors (ever) to discard and all new subscription requests
to require moderation.
Sorry for the noise.
Bill
--
Bill Mills
Principal Technical Consultant, Linaro
+1-240-643-0836
TZ: US Eastern
Work Schedule: Tues/Wed/Thur
Hi All,
This email comes from the github PR but I am going to respond here.
On 2/23/26 4:45 AM, bertrand-marquis wrote:
> Hi Everyone,
>
> This pull request include a number of fixes or changes to handle the review
> comment we got on virtio mailing list.
>
> Some points are not handled because they need a decision between us first
> (following summary was generated by chatgpt):
>
> Ordering / Out‑of‑Order Feature Discussion
>
> * Parav Pandit proposes relaxing the strict request/response ordering
> because it is too constraining for high‑performance data paths. He
> suggests that the bus may need to deliver responses out of
> order, and the spec should support that for performance.
> * Demi Marie Obenour raises reset and out‑of‑order race risks if ordering
> is relaxed. She suggests a generation/stream number or bus‑level reset
> semantics to avoid stale or mis‑matched responses.
> * Michael S. Tsirkin recommends keeping v1 simple and introducing
> performance features via optional feature bits. This aligns with making
> relaxed ordering a negotiated capability rather than a
> baseline requirement.
>
I don't think transport level operations (excepting Virtqueue notifications
) have a big impact on performance. Demi seemed to understand our POV once
she saw the option for events to go out of band.
At least part of the time Parav was looking at virtio-msg NOT as a
transport replacement (using existing virtqueue's in shared memory) and
instead was looking at it as full virtio over a message channel (like
virtio over tcp). Michael seemed to agree that was out of scope for our
work and would need to be its own thing. However it is not clear if or
when Parav has changed his thinking.
> Configuration Generation Optional Feature Discussion
>
> * Peter Hilber suggests making the SET_CONFIG generation count optional
> or dropping it, because Linux drivers assume config writes are not
> rejected and do not retry. His rationale is avoiding
> incompatibility with existing driver behavior.
TL;DR include the generation counter in the message but eliminate the
"rejection" of the SET_CONFIG and make it the devices job to coalesce the
new value into the config space.
virtio-msg has a race condition in config write that virtio-pci and
virtio-mmio does not have due to its trap and emulate nature. In
virtio-msg a write can race with a device side change to the same word(s).
I agree with Peter that we should not expect the individual Linux drivers
to respond to this. My initial thought was to do this in the driver side
transport but Peter made we realize this is dumb. The transport will not
have any extra info. It is better to push this race resolution back to the
device. Giving the device the generation count allows the device to know
if the driver was modifying the old value or the new value.
I fully expect many device side implementations to ignore the generation
count and just to take the new value but providing the generation count
allows it to be more sophisticated if it wants to be.
>
> VQ Enabled Flag Discussion
>
> * Peter Hilber suggests adding an explicit “queue enabled” indication in
> SET_VQUEUE (and also reporting it in GET_VQUEUE) so drivers can avoid
> an extra GET_VQUEUE round‑trip just to confirm
> enablement. This is aimed at reducing overhead in the initialization flow.
>
Ack
> Device Number Size Discussion
>
> * Parav Pandit argues the 16‑bit device number is too small; he suggests
> 32‑bit to cover >64k devices and PCI BDF domain usage.
> * Demi Marie Obenour argues device identifiers should be 64‑bit or even
> 128‑bit to ensure long‑term uniqueness.
> * Peter Hilber highlights that reuse of device numbers after removal can
> cause races, and suggests adding a reuse‑limiting policy. He ties this
> to the broader question of expanding the device number
> space.
>
32 or 64 is fine. 128 seems overkill yes? One PCI bus has a limit of 2^64
devices.
> EVENT_CONFIG Data Mandatory Discussion
>
> * Demi Marie Obenour proposes making EVENT_CONFIG always include
> configuration data (MUST), to avoid a round trip. If data is mandatory,
> she notes that offset/length‑only signaling becomes
> unnecessary.
NAK. This will prevent simple out of band notification for the EVENT_AVAIL.
Demi seemed to want OOB notifications so this seems in contradiction.
> * There is a noted contradiction with earlier guidance (Peter Hilber) and
> the current spec change ( I already updated the spec to require offset/
> length even when data is omitted,
Even this will prevent OOB. The existing mechanisum allow this to be
negociated. Any bus that uses OOB can NAK these feature bits to prevent use.
> keeping data optional as I think this
> makes sense and i am not quite sure we can mandate the data without
> breaking compatibility with other transports or existing devices). This
> needs a clear decision.
>
> ---------------------------------------------------------------------------
>
>
> You can view, comment on, or merge this pull request online at:
>
> https://github.com/Linaro/virtio-msg-spec/pull/31 <https://github.com/
> Linaro/virtio-msg-spec/pull/31>
>
>
> Commit Summary
>
> * 52ffe64 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/52ffe641810d26860fbdbb45e7437c8df9360f6a> virtio-msg: clarify
> device number vs device ID (Peter Hilber)
> * 3a70c3a <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/3a70c3ad6a277cf08931a4efb86b3a41fd29eb7b> virtio-msg: fix minor
> grammar in device topology (Peter Hilber)
> * d677fc8 <https://github.com/Linaro/virtio-msg-spec/pull/31/commits/
> d677fc8c1717320416460d911143fa243b8f4eb8> virtio-msg: use “transport
> revision” consistently (Peter Hilber)
> * c0020f1 <https://github.com/Linaro/virtio-msg-spec/pull/31/commits/
> c0020f12b985b25c66e3afcf348c1b7aaa81fb78> virtio-msg: use active voice
> for reserved header bits (Peter Hilber)
> * 7eafee2 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/7eafee2d67d847862f6ff7a9a2899fb5b081b217> virtio-msg: drop
> duplicate GET_DEVICE_INFO requirement (Peter Hilber)
> * 623334c <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/623334cf0ae1c2289d23c3b47bdd7f2cfa801ed7> virtio-msg: clarify
> removal handling (Peter Hilber)
> * 4c75e07 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/4c75e07c29f4aa2dbdccfcde8c89d7d9953f28f6> virtio-msg: allow
> zero devices per bus instance (Peter Hilber)
> * 2bbad99 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/2bbad994805f216755ba6abfe3a1b844e41f30aa> virtio-msg: add note
> on config vs virtqueue data (Demi Marie Obenour)
> * 094111b <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/094111b9d9e019c9cc4e50d31bf1c27ea5c8d3dc> virtio-msg: rename
> READY to ADDED in bus device events (Parav Pandit)
> * 1e1e817 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/1e1e81778404d596f1216385a8c8945cf6dbdbdc> virtio-msg: clarify
> transport feature negotiation (Peter Hilber)
> * f50fe3c <https://github.com/Linaro/virtio-msg-spec/pull/31/commits/
> f50fe3c985bd6061449f7ab50b45a045bbe15aa7> virtio-msg: clarify
> generation count reset semantics (Peter Hilber)
> * f462169 <https://github.com/Linaro/virtio-msg-spec/pull/31/commits/
> f462169090b0907b59b14ed0b6ee952c67934df1> virtio-msg: clarify
> notification wording (Peter Hilber)
> * 2d8c995 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/2d8c9952035cf72331de5b22ea13aacd9b2c5a29> virtio-msg: require
> EVENT_USED unless polling (Peter Hilber)
> * 4b8c483 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/4b8c483c79709a509a5961309435a941a8e17110> virtio-msg: drop
> RESET_VQUEUE-before-reprogramming (Peter Hilber)
> * b783fc5 <https://github.com/Linaro/virtio-msg-spec/pull/31/commits/
> b783fc534d8973167770515d12a5fd22703ba331> virtio-msg: align admin vq
> field widths (Peter Hilber)
> * 818142e <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/818142ed4c8612645a46ed211a0a172267684d35> virtio-msg: proposal:
> clarify echoed fields (Peter Hilber)
> * 6ba8c70 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/6ba8c708335f9068c0045f74714f9258b8faf488> virtio-msg: require
> clearing FEATURES_OK (Peter Hilber)
> * 21aeb49 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/21aeb4982be32a80ea2a4852cef9dae8f552786d> virtio-msg: widen
> GET_SHM fields (Peter Hilber)
> * 3e007fd <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/3e007fdbc75eadb931c2cb4eaf4954255f51e419> virtio-msg: clarify
> EVENT_CONFIG ranges (Peter Hilber)
> * f109559 <https://github.com/Linaro/virtio-msg-spec/pull/31/commits/
> f1095590057baff76976d371a1ecd95357ca3f35> virtio-msg: streamline
> DEVICE_NEEDS_RESET text (Peter Hilber)
> * 835ef71 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/835ef718e432d7b8df08cf0e25f59f89a0a0bdc6> virtio-msg: scope
> msg_id validation (Peter Hilber)
> * 07b0d08 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/07b0d08d12c9038d59f07a627dbf408ec7424435> virtio-msg: drop
> FEATURES_OK sentence (Peter Hilber)
> * ba844e9 <https://github.com/Linaro/virtio-msg-spec/pull/31/commits/
> ba844e93dd1a685fae0d0c04eb216784962a91e1> virtio-msg: note bus-level
> errors (Demi Marie Obenour)
> * 3c685b6 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/3c685b6d5facca7ff01cce29869893c077153209> virtio-msg: reframe
> ordering responses (Demi Marie Obenour)
> * 8a529c0 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/8a529c0a6bd261ef0bdda92b1715bfb5e5e8e034> virtio-msg: clarify
> virtqueue data path (Demi Marie Obenour)
> * 0df8574 <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/0df8574759c0685e802a99c34ef09e0d7cef3af2> virtio-msg: allow
> out-of-order events (Demi Marie Obenour)
> * 773970f <https://github.com/Linaro/virtio-msg-spec/pull/31/
> commits/773970f939facdba7ba7f2ff70838d0133e6cd58> virtio-msg: use
> physical address terminology (Demi Marie Obenour)
>
>
> File Changes
>
> (1 file <https://github.com/Linaro/virtio-msg-spec/pull/31/files>)
>
> * *M* transport-msg.tex <https://github.com/Linaro/virtio-msg-spec/
> pull/31/files#diff-
> ae43224efc56fefe20a4d5de6f7cfb5ccd94950be1246b6ece5274c6c994e167> (232)
>
>
> Patch Links:
>
> * https://github.com/Linaro/virtio-msg-spec/pull/31.patch <https://
> github.com/Linaro/virtio-msg-spec/pull/31.patch>
> * https://github.com/Linaro/virtio-msg-spec/pull/31.diff <https://
> github.com/Linaro/virtio-msg-spec/pull/31.diff>
>
> —
> Reply to this email directly, view it on GitHub <https://github.com/Linaro/
> virtio-msg-spec/pull/31>, or unsubscribe <https://github.com/notifications/
> unsubscribe-auth/
> AIBXWC5SZFXJ2MVGMCE7AYL4NLD5PAVCNFSM6AAAAACV4PIWDWVHI2DSMVQWIX3LMV43ASLTON2WKOZTHE3TOMRUG42TEMI>.
> You are receiving this because your review was requested.Message ID:
> <Linaro/virtio-msg-spec/pull/31(a)github.com>
>
Hi Everyone,
I pushed a first pull request to try to handle some of the comments we got on virtio mailing list:
https://github.com/Linaro/virtio-msg-spec/pull/31
In my comment i list some of the points for which we will need to have a discussion on thursday to decide how and if we handle them.
Please check and add any comments you might have.
Cheers
Bertrand
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hello,
I am trying to get feedback / ideas (specially from the people in
--to) before I reach out to LKML.
Problem statement:
Share memory (dma-buf/dma-heap fd) over VSock (over virtio-msg-ffa).
Current status:
I have implemented the first version [1]/[2]. The system-heap memory
is successfully shared with OPTEE, test setup available here [3].
Workflow:
Userspace:
open("/dev/dma_heap/system", O_RDWR)
+
ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc)
+
sendmsg(sock, &msg, 0)
// msg: SCM_VSOCK_SHMEM / VSOCK_SHMEM_TYPE_FFA / VSOCK_SHMEM_SUBOP_OFFER
Kernel:
vsock_sendmsg_shmem(vsk, msg)
|
vsk->transport->map_dma_buf(dmabuf)
|
virtio_transport_map_dma_buf(dmabuf)
|
dma_buf_attach(dmabuf, dev); // Vsock's underlying device - virtio-msg device
+
dma_buf_map_attachment_unlocked(dbuf->attach, DMA_BIDIRECTIONAL);
|
system_heap_map_dma_buf(attach, dir);
|
dma_map_sgtable(attachment->dev, table, dir, attrs=0);
|
virtio_msg_dma_map_sg_dev(dev, sgl, nents, dir, attrs); // FFA dma-ops
|
ffa_dev->ops->mem_ops->memory_share(&args);
+
vmfdev->send(vmfdev, request, response, &vmfdev->idata);
The memory is shared with FFA SHARE type. This doesn't require a
custom heap and works with any DMA heap/buf, system heap is just an
example here.
FFA supports LEND-ing the memory as well and Google needs it. And this
is where I am _blocked_ right now and need help.
We need to support both SHARE (both sides can access memory) and LEND
(Linux doesn't access it any further - unmapped) mechanisms and I am
not sure how to move forward.
- We can pass the share-type from userspace and that can reach up to
vsock / dma-buf (with minor API change to
dma_buf_map_attachment_unlocked()) / system-heap.
- But after that the standard dma_map_sgtable() call takes only one
configurable parameter, `attributes` (which is set to 0 for now by
system heap). There is nothing else which I can use to pass the flag
to FFA specific DMA OPs.
- I thought maybe I can create a new DMA attribute:
`DMA_ATTR_MEM_LEND`, which can be passed in the `attr` field and
that will work I guess.
- But I am not sure if it would be acceptable upstream to add a new
attribute for this.
- And so I am looking for some ideas.
--
viresh
[1] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=vi…
[2] Patches of interest
- vsock: Add shared-memory support for virtio and loopback
- virtio-msg: ffa: Add device DMA-OPS
[3] https://linaro.atlassian.net/wiki/spaces/HVAC/pages/30372528129
This series adds the virtio-msg transport layer.
The individuals and organizations involved in this effort have had difficulty in
using the existing virtio-transports in various situations and desire to add one
more transport that performs its transport layer operations by sending and
receiving messages.
Implementations of virtio-msg will normally be done in multiple layers:
* common / device level
* bus level
The common / device level defines the messages exchanged between the driver
and a device. This common part should lead to a common driver holding most
of the virtio specifics and can be shared by all virtio-msg bus implementations.
The kernel implementation in [3] shows this separation. As with other transport
layers, virtio-msg should not require modifications to existing virtio device
implementations (virtio-net, virtio-blk etc). The common / device level is the
main focus of this version of the patch series.
The virtio-msg bus level implements the normal things a bus defines
(enumeration, dma operations, etc) but also implements the message send and
receive operations. A number of bus implementations are envisioned,
some of which will be reusable and general purpose. Other bus implementations
might be unique to a given situation, for example only used by a PCIe card
and its driver.
How much of the bus level should be described in the virtio spec is one item
we wish to discuss. This draft takes a middle approach by describing the bus
level and defining some standard bus level messages that MAY be used by the bus.
It also describes a range of bus messages that are implementation dependent.
The standard bus messages are an effort to avoid different bus implementations
doing the same thing in different ways for no good reason. However the
different environments will require different things. Instead of trying to
anticipate all needs and provide something very abstract, we think
implementation specific messages will be needed at the bus level. Over time,
if we see similar messages across multiple bus implementations, we will move to
standardize a bus level message for that.
We are working on a few reusable bus implementations:
* virtio-msg-ffa based on Arm FF-A interface for use between:
* normal world and secure world
* host and VM or VM to VM
* Can be used w/ or with out a hypervisor
* Any Hypervisor that implements FF-A can be used
* We have this working with pKVM and Xen
* We have this working with Trusty and OP-TEE
* virtio-msg-amp for use between heterogenous systems
* The main processors and its co-processors on an AMP SOC
* Two or more systems connected via PCIe
* Minimal requirements: bi-directional interrupts and
at least one shared memory area
* hvac-demo has 2 demos of this
* This is working on two hardware platforms
* virtio-msg-loopback for userspace implemented devices
* Allows user space to provide devices to its own kernel
* This is similar to fuse, cuse or loopback block devices but for virtio
* hvac-demo has a demo
We also anticipate a few more:
* virtio-msg-xen specific to Xen
* Usable on any Xen system (including x86 where FF-A does not exist)
* Using Xen events and page grants
* virtio-msg over admin virtqueues
* This allows any virtio-pci device that supports admin virtqueues to also
support a virtio-msg bus that supports sub devices
* [We are looking for collaborators for this work]
Changes since RFC2:
Spec Functional:
* Made the common message header 8 bytes and added a token for optional use
when parallel outstanding requests are possible
* Made the 8 byte fields align to 8 byte offsets.
This effects the {SET,GET}_VQUEUE messages
* Many conformance cases have been tightened.
Spec Editorial:
* Major restructure to better align with virtio spec
* Conformance model now matches other transports
* Use C structures to define messages instead of tables
* Added a section to describe how responses are matched to requests
This includes the use of the new token field
* Redefined / better described error handling between transport and bus
layers to eliminate the need for the bus to generate fake response messages
* Included editorial feedback from RFC2
Eco-system:
* Added virtio-msg-loopback demo
* Arm has published the first draft of the virtio-msg over FFA spec [6]
* virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE
secure world
* LKML RFC has been sent [7]
* QEMU RFC has been sent [8]
This is the first non-RFC patch series. The known short comings have been
addressed. We ask for review in earnest on this series and thank you for
any feedback you can provide.
Background info and work in progress implementations:
* HVAC project page with intro slides [1]
* HVAC demo repo w/ instructions in README.md [2]
* Kernel w/ virtio-msg common level and ffa support [3]
* QEMU w/ support for one form of virtio-msg-amp [4]
* Portable RTOS library w/ one form of virtio-msg-amp [5]
In addition to the QEMU system based demos in the hvac-demo repo, we also have
two hardware systems running:
* AMD x86 + AMD Arm Versal connected via PCIe
* ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr
Please note that although the demos work, a few have not been aligned
with this version of the spec.
[1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview
[2] https://github.com/wmamills/hvac-demo
[3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=vi…
[4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-rfc/
[5] https://github.com/arnopo/open-amp/commits/virtio-msg/
[6] https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c
[7] https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linaro.org/
[8] https://mail.gnu.org/archive/html/qemu-devel/2025-10/msg07438.html
Bertrand Marquis (2):
virtio-msg: add new command for bus normative
virtio-msg: add conformance entries in conformance chapter
Bill Mills (2):
virtio-msg: Add virtio-msg, a message based virtio transport layer
virtio-msg: link virtio-msg content
commands.tex | 3 +-
conformance.tex | 105 ++-
content.tex | 1 +
transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 1746 insertions(+), 3 deletions(-)
create mode 100644 transport-msg.tex
--
2.34.1
This series adds the virtio-msg transport layer.
The individuals and organizations involved in this effort have had difficulty in
using the existing virtio-transports in various situations and desire to add one
more transport that performs its transport layer operations by sending and
receiving messages.
Implementations of virtio-msg will normally be done in multiple layers:
* common / device level
* bus level
The common / device level defines the messages exchanged between the driver
and a device. This common part should lead to a common driver holding most
of the virtio specifics and can be shared by all virtio-msg bus implementations.
The kernel implementation in [3] shows this separation. As with other transport
layers, virtio-msg should not require modifications to existing virtio device
implementations (virtio-net, virtio-blk etc). The common / device level is the
main focus of this version of the patch series.
The virtio-msg bus level implements the normal things a bus defines
(enumeration, dma operations, etc) but also implements the message send and
receive operations. A number of bus implementations are envisioned,
some of which will be reusable and general purpose. Other bus implementations
might be unique to a given situation, for example only used by a PCIe card
and its driver.
How much of the bus level should be described in the virtio spec is one item
we wish to discuss. This draft takes a middle approach by describing the bus
level and defining some standard bus level messages that MAY be used by the bus.
It also describes a range of bus messages that are implementation dependent.
The standard bus messages are an effort to avoid different bus implementations
doing the same thing in different ways for no good reason. However the
different environments will require different things. Instead of trying to
anticipate all needs and provide something very abstract, we think
implementation specific messages will be needed at the bus level. Over time,
if we see similar messages across multiple bus implementations, we will move to
standardize a bus level message for that.
We are working on a few reusable bus implementations:
* virtio-msg-ffa based on Arm FF-A interface for use between:
* normal world and secure world
* host and VM or VM to VM
* Can be used w/ or with out a hypervisor
* Any Hypervisor that implements FF-A can be used
* We have this working with pKVM and Xen
* We have this working with Trusty and OP-TEE
* virtio-msg-amp for use between heterogenous systems
* The main processors and its co-processors on an AMP SOC
* Two or more systems connected via PCIe
* Minimal requirements: bi-directional interrupts and
at least one shared memory area
* hvac-demo has 2 demos of this
* This is working on two hardware platforms
* virtio-msg-loopback for userspace implemented devices
* Allows user space to provide devices to its own kernel
* This is similar to fuse, cuse or loopback block devices but for virtio
* hvac-demo has a demo
We also anticipate a few more:
* virtio-msg-xen specific to Xen
* Usable on any Xen system (including x86 where FF-A does not exist)
* Using Xen events and page grants
* virtio-msg over admin virtqueues
* This allows any virtio-pci device that supports admin virtqueues to also
support a virtio-msg bus that supports sub devices
* [We are looking for collaborators for this work]
Changes since RFC2:
Spec Functional:
* Made the common message header 8 bytes and added a token for optional use
when parallel outstanding requests are possible
* Made the 8 byte fields align to 8 byte offsets.
This effects the {SET,GET}_VQUEUE messages
* Many conformance cases have been tightened.
Spec Editorial:
* Major restructure to better align with virtio spec
* Conformance model now matches other transports
* Use C structures to define messages instead of tables
* Added a section to describe how responses are matched to requests
This includes the use of the new token field
* Redefined / better described error handling between transport and bus
layers to eliminate the need for the bus to generate fake response messages
* Included editorial feedback from RFC2
Eco-system:
* Added virtio-msg-loopback demo
* Arm has published the first draft of the virtio-msg over FFA spec [6]
* virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE
secure world
* LKML RFC has been sent [7]
* QEMU RFC has been sent [8]
This is the first non-RFC patch series. The known short comings have been
addressed. We ask for review in earnest on this series and thank you for
any feedback you can provide.
Background info and work in progress implementations:
* HVAC project page with intro slides [1]
* HVAC demo repo w/ instructions in README.md [2]
* Kernel w/ virtio-msg common level and ffa support [3]
* QEMU w/ support for one form of virtio-msg-amp [4]
* Portable RTOS library w/ one form of virtio-msg-amp [5]
In addition to the QEMU system based demos in the hvac-demo repo, we also have
two hardware systems running:
* AMD x86 + AMD Arm Versal connected via PCIe
* ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr
Please note that although the demos work, a few have not been aligned
with this version of the spec.
[1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview
[2] https://github.com/wmamills/hvac-demo
[3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=vi…
[4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-rfc/
[5] https://github.com/arnopo/open-amp/commits/virtio-msg/
[6] https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c
[7] https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linaro.org/
[8] https://mail.gnu.org/archive/html/qemu-devel/2025-10/msg07438.html
Alex Bennée (1):
local: add local build help and github automation (!UPSTREAM)
Bertrand Marquis (2):
virtio-msg: add new command for bus normative
virtio-msg: add conformance entries in conformance chapter
Bill Mills (5):
email: Add script and cover letter for RFC1
email: adjust cover letter for RFC2
email: adjust cover letter for RFC3
virtio-msg: Add virtio-msg, a message based virtio transport layer
virtio-msg: link virtio-msg content
.github/PULL_REQUEST_TEMPLATE.md | 14 +-
.github/workflows/deploy.yaml | 38 +
.github/workflows/test.yml | 15 +
.prjinfo/sendmail/cover.txt | 125 +++
.prjinfo/sendmail/send.sh | 57 ++
Makefile | 33 +
REVISION | 2 +-
commands.tex | 3 +-
conformance.tex | 105 +-
content.tex | 1 +
make-setup-generated.sh | 31 +-
makeall.sh | 4 +-
makehtml.sh | 2 +-
makepdf.sh | 2 +-
transport-msg.tex | 1640 ++++++++++++++++++++++++++++++
15 files changed, 2052 insertions(+), 20 deletions(-)
create mode 100644 .github/workflows/deploy.yaml
create mode 100644 .github/workflows/test.yml
create mode 100644 .prjinfo/sendmail/cover.txt
create mode 100755 .prjinfo/sendmail/send.sh
create mode 100644 Makefile
create mode 100644 transport-msg.tex
--
2.34.1
This series adds the virtio-msg transport layer.
The individuals and organizations involved in this effort have had difficulty in
using the existing virtio-transports in various situations and desire to add one
more transport that performs its transport layer operations by sending and
receiving messages.
Implementations of virtio-msg will normally be done in multiple layers:
* common / device level
* bus level
The common / device level defines the messages exchanged between the driver
and a device. This common part should lead to a common driver holding most
of the virtio specifics and can be shared by all virtio-msg bus implementations.
The kernel implementation in [3] shows this separation. As with other transport
layers, virtio-msg should not require modifications to existing virtio device
implementations (virtio-net, virtio-blk etc). The common / device level is the
main focus of this version of the patch series.
The virtio-msg bus level implements the normal things a bus defines
(enumeration, dma operations, etc) but also implements the message send and
receive operations. A number of bus implementations are envisioned,
some of which will be reusable and general purpose. Other bus implementations
might be unique to a given situation, for example only used by a PCIe card
and its driver.
How much of the bus level should be described in the virtio spec is one item
we wish to discuss. This draft takes a middle approach by describing the bus
level and defining some standard bus level messages that MAY be used by the bus.
It also describes a range of bus messages that are implementation dependent.
The standard bus messages are an effort to avoid different bus implementations
doing the same thing in different ways for no good reason. However the
different environments will require different things. Instead of trying to
anticipate all needs and provide something very abstract, we think
implementation specific messages will be needed at the bus level. Over time,
if we see similar messages across multiple bus implementations, we will move to
standardize a bus level message for that.
We are working on a few reusable bus implementations:
* virtio-msg-ffa based on Arm FF-A interface for use between:
* normal world and secure world
* host and VM or VM to VM
* Can be used w/ or with out a hypervisor
* Any Hypervisor that implements FF-A can be used
* We have this working with pKVM and Xen
* We have this working with Trusty and OP-TEE
* virtio-msg-amp for use between heterogenous systems
* The main processors and its co-processors on an AMP SOC
* Two or more systems connected via PCIe
* Minimal requirements: bi-directional interrupts and
at least one shared memory area
* hvac-demo has 2 demos of this
* This is working on two hardware platforms
* virtio-msg-loopback for userspace implemented devices
* Allows user space to provide devices to its own kernel
* This is similar to fuse, cuse or loopback block devices but for virtio
* hvac-demo has a demo
We also anticipate a few more:
* virtio-msg-xen specific to Xen
* Usable on any Xen system (including x86 where FF-A does not exist)
* Using Xen events and page grants
* virtio-msg over admin virtqueues
* This allows any virtio-pci device that supports admin virtqueues to also
support a virtio-msg bus that supports sub devices
* [We are looking for collaborators for this work]
Changes since RFC2:
Spec Functional:
* Made the common message header 8 bytes and added a token for optional use
when parallel outstanding requests are possible
* Made the 8 byte fields align to 8 byte offsets.
This effects the {SET,GET}_VQUEUE messages
Spec Editorial:
* Major restructure to better align with virtio spec
* Conformance model now matches
* Use C structures to define messages instead of tables
* Added a section to describe how responses are matched to requests
This includes the use of the new token field
* Redefined / better described error handling between transport and bus
layers to eliminate the need for the bus to generate fake response messages
* Included editorial feedback from RFC2
Eco-system:
* Add virtio-msg-loopback demo
* Arm has published the first draft of the virtio-msg over FFA spec
* virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE
secure world
* QEMU RFCv1 has been sent
* LKML RFCs have been sent
Background info and work in progress implementations:
* HVAC project page with intro slides [1]
* HVAC demo repo w/ instructions in README.md [2]
* Kernel w/ virtio-msg common level and ffa support [3]
* QEMU w/ support for one form of virtio-msg-amp [4]
* Portable RTOS library w/ one form of virtio-msg-amp [5]
In addition to the QEMU system based demos in the hvac-demo repo, we also have
two hardware systems running:
* AMD x86 + AMD Arm Versal connected via PCIe
* ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr
Please note that although the demos work, not all of them have been aligned
with this version of the spec.
[1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview
[2] https://github.com/wmamills/hvac-demo
[3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=vi…
[4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-new
[5] https://github.com/arnopo/open-amp/commits/virtio-msg/
Bertrand Marquis (2):
virtio-msg: add new command for bus normative
virtio-msg: add conformance entries in conformance chapter
Bill Mills (2):
virtio-msg: Add virtio-msg, a message based virtio transport layer
virtio-msg: link virtio-msg content
commands.tex | 3 +-
conformance.tex | 105 ++-
content.tex | 1 +
transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 1746 insertions(+), 3 deletions(-)
create mode 100644 transport-msg.tex
--
2.34.1
Happy new year everyone,
I have a few questions regarding the design of the whole memory sharing over
Vsock work and looking to get some feedback before I spend time making the
changes.
The initial design (from Nov):
- Initially I implemented a FFA based DMA heap (no DMA ops used), which would
allocate memory and make direct FFA calls to send the memory to the other
endpoint.
- The userspace would open this heap, allocated dma-buf and pass its FD over
Vsock.
- The Vsock layer would then call ->shmem() (a new callback I added to
dma-buf/heap, which will send memory over FFA and return ffa bus address), to
get the metadata to be shared over Vsock.
The current design:
- In one of the calls Bertrand suggested to not create parallel paths for
sending memory over FFA. Instead we should use the exiting dma-ops for FFA
(used for virtqueue and reserved-mem) somehow.
- I created a platform device (as a child of the FFA device) and assigned a new
set of DMA ops to it (the only difference from reserved-mem ops is that we
don't do swiotlb here and allocate fresh instead of the reserved mem).
- This pdev is used by the DMA heap to allocate memory using
dma_alloc_coherent() and that made sure everything got mapped correctly.
- I still need a dma-buf helper to get the metadata to send over Vsock. The
existing helper was renamed as s/shmem/shmem_data.
The future design (that I have questions about):
- The FFA specific DMA heap I now have doesn't do anything special compared to
the system heap, mostly exactly same.
- Which made me realize that probably I shouldn't add a new heap (Google can add
one later if they really want) and the solution should work with any heap /
dma-buf.
- So, userspace should allocate heap from system-heap, get a dma-buf from it and
send its FD.
- The vsock layer then should attach this dma-buf to a `struct device` somehow
and then call map_dma_buf() for the dma-buf. This requires the dma-ops of the
device to be set to the FFA based dma-ops and then it should just work.
- The tricky point is finding that device struct (as Vsock can't get it from
dma-buf or usersapce).
- One way, I think (still needs exploring but should be possible) is to use the
struct device of the virtio-msg device over which Vsock is implemented. We can
set the dma-ops of the virtio-msg device accordingly.
- The system heap doesn't guarantee contiguous allocation though (which my FFA
heap did) and so we will be required to send a scatter-gather list over vsock,
instead of just one address and size (what I am doing right now).
- Does this make sense ? Or if there is a use-case that this won't solve, etc ?
--
viresh
Hello,
I have some questions / feedback on the Xen side of virtio-msg.
AFAIU virtio-msg defines a protocol in order to deal with discovery
(optional) and configuration of the PV devices. But things are undefined
regarding what is a "memory address".
In Xen memory model with grants, each guest has its own memory space.
The frontend shares memory pages with the backend through grants pages
identified by grant references.
So in a design based on grants, virtio addresses can't (or at least
shouldn't) be guest physical address; but needs to be something derived
on grants. A earlier design [1] was forging a address with the grant
reference, but I feel it's not great, as it forces "map+unmap" cycles
for temporary buffers thus has the same performance problem as Xen PV
drivers (without persistent grants) where map+unmap cycles is a
performance problem.
And would make the address space very fragmented and in often limited to
4KB buffers.
One idea that is already used by Xen displif (PV Display) is to have a
"gref directory" and describe the address space on that. The gref
directory is roughly a array of grant references (shared pages infos)
that could describe a address space starting at 0, where each page is
defined by a grant reference of the directory. That way, the backend can
freely keep all or a part or the address space persistently mapped (or
eventually map it all at once); and the address space is also contiguous
which would help with >4KB buffers.
Any thoughts ?
[1]
https://static.sched.com/hosted_files/xen2021/bf/Thursday_2021-Xen-Summit-v…
--
Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
