On 10/16/26 22:26, Karl Mehltretter wrote:
On Tue, Feb 17, 2026 at 08:11:34PM -0400, Jason Gunthorpe wrote:
- match_args.attach = attach;
- if (dmabuf->ops->match_mapping) {
ret = dmabuf->ops->match_mapping(&match_args);- }
- attach->dmabuf = dmabuf; attach->importer_ops = importer_ops; attach->importer_priv = importer_priv;
I may misunderstand the intended contract. match_mapping() runs while the attachment is still zeroed and without dmabuf->resv held.
Patch 11's amdgpu callback needs importer_ops and importer_priv, so its XGMI match cannot work here. Patch 12's VFIO callback asserts dmabuf->resv and reads priv->vdev, which cleanup changes under that lock.
Should the core initialize the attachment and hold dmabuf->resv for match_mapping(), or should callbacks avoid attachment and lock-protected state?
Right. This completely misplaced.
The matching can't be done at attachment time, but rather needs to be done before creating a mapping while holding the resv lock so that the matching decision can't change.
Regards, Christian.
Thanks, Karl
On Mon, 2026-10-19 at 10:35 +0200, Christian König wrote:
On 10/16/26 22:26, Karl Mehltretter wrote:
On Tue, Feb 17, 2026 at 08:11:34PM -0400, Jason Gunthorpe wrote:
- match_args.attach = attach;
- if (dmabuf->ops->match_mapping) {
ret = dmabuf->ops->match_mapping(&match_args);- }
attach->dmabuf = dmabuf; attach->importer_ops = importer_ops; attach->importer_priv = importer_priv;
I may misunderstand the intended contract. match_mapping() runs while the attachment is still zeroed and without dmabuf->resv held.
Patch 11's amdgpu callback needs importer_ops and importer_priv, so its XGMI match cannot work here. Patch 12's VFIO callback asserts dmabuf-
resv and
reads priv->vdev, which cleanup changes under that lock.
Should the core initialize the attachment and hold dmabuf->resv for match_mapping(), or should callbacks avoid attachment and lock- protected state?
Right. This completely misplaced.
The matching can't be done at attachment time, but rather needs to be done before creating a mapping while holding the resv lock so that the matching decision can't change.
So question is what exactly should the contract be? Looking at the pcie_p2p negotiation today it looks somewhat like.
1) Importer and exporter agree at attach time what mappings are allowed pending placement conditions and other attachers. 2) At map-time the mapping is negotiated taking all constraints into account. 3) The negotiated mapping is valid until an invalidate_mappings() callback is received.
Following this would allow a re-implementation of pcie_p2p as a mapping type and would allow a controlled link-severing to send an invalidate_mappings() notifier and the mappings to be re-negotiated.
Importers requiring an early hint as to what the actual mapping would look like would need to be prepared to re-consider at map-time, with a reasonable assumption that the reconsideration and re-mapping would happen seldom enough.
Thanks, Thomas
Regards, Christian.
Thanks, Karl
On Mon, Oct 19, 2026 at 02:43:29PM +0200, Thomas Hellström wrote:
On Mon, 2026-10-19 at 10:35 +0200, Christian König wrote:
On 10/16/26 22:26, Karl Mehltretter wrote:
On Tue, Feb 17, 2026 at 08:11:34PM -0400, Jason Gunthorpe wrote:
- match_args.attach = attach;
- if (dmabuf->ops->match_mapping) {
ret = dmabuf->ops->match_mapping(&match_args);- }
attach->dmabuf = dmabuf; attach->importer_ops = importer_ops; attach->importer_priv = importer_priv;
I may misunderstand the intended contract. match_mapping() runs while the attachment is still zeroed and without dmabuf->resv held.
Patch 11's amdgpu callback needs importer_ops and importer_priv, so its XGMI match cannot work here. Patch 12's VFIO callback asserts dmabuf-
resv and
reads priv->vdev, which cleanup changes under that lock.
Should the core initialize the attachment and hold dmabuf->resv for match_mapping(), or should callbacks avoid attachment and lock- protected state?
Right. This completely misplaced.
The matching can't be done at attachment time, but rather needs to be done before creating a mapping while holding the resv lock so that the matching decision can't change.
So question is what exactly should the contract be? Looking at the pcie_p2p negotiation today it looks somewhat like.
- Importer and exporter agree at attach time what mappings are allowed
pending placement conditions and other attachers. 2) At map-time the mapping is negotiated taking all constraints into account. 3) The negotiated mapping is valid until an invalidate_mappings() callback is received.
Following this would allow a re-implementation of pcie_p2p as a mapping type and would allow a controlled link-severing to send an invalidate_mappings() notifier and the mappings to be re-negotiated.
Importers requiring an early hint as to what the actual mapping would look like would need to be prepared to re-consider at map-time, with a reasonable assumption that the reconsideration and re-mapping would happen seldom enough.
The idea was the supported mapping types are fixed properties of the importer and exporter.
Like iommufd is not suddenly going to start understanding ualink after an invalidation.
Further, I had the (perhaps naive) expectation that the interconnect option was not something that would change during mapping. I did not think you would loose your PCIe option if ualink becomes available, or vice versa. Though on deeper reflection I suppose someone could build HW so you might loose your ualink option if the dmabuf is moved to system memory while PCIe would still work.
Making the decision right before map seems like a reasonable enhancement, but I can no longer remember off hand if that would cause complications :)
Jason
On Mon, 2026-10-19 at 10:30 -0300, Jason Gunthorpe wrote:
On Mon, Oct 19, 2026 at 02:43:29PM +0200, Thomas Hellström wrote:
On Mon, 2026-10-19 at 10:35 +0200, Christian König wrote:
On 10/16/26 22:26, Karl Mehltretter wrote:
On Tue, Feb 17, 2026 at 08:11:34PM -0400, Jason Gunthorpe wrote:
- match_args.attach = attach;
- if (dmabuf->ops->match_mapping) {
ret = dmabuf->ops-match_mapping(&match_args);
- }
attach->dmabuf = dmabuf; attach->importer_ops = importer_ops; attach->importer_priv = importer_priv;
I may misunderstand the intended contract. match_mapping() runs while the attachment is still zeroed and without dmabuf->resv held.
Patch 11's amdgpu callback needs importer_ops and importer_priv, so its XGMI match cannot work here. Patch 12's VFIO callback asserts dmabuf-
resv and
reads priv->vdev, which cleanup changes under that lock.
Should the core initialize the attachment and hold dmabuf->resv for match_mapping(), or should callbacks avoid attachment and lock- protected state?
Right. This completely misplaced.
The matching can't be done at attachment time, but rather needs to be done before creating a mapping while holding the resv lock so that the matching decision can't change.
So question is what exactly should the contract be? Looking at the pcie_p2p negotiation today it looks somewhat like.
- Importer and exporter agree at attach time what mappings are
allowed pending placement conditions and other attachers. 2) At map-time the mapping is negotiated taking all constraints into account. 3) The negotiated mapping is valid until an invalidate_mappings() callback is received.
Following this would allow a re-implementation of pcie_p2p as a mapping type and would allow a controlled link-severing to send an invalidate_mappings() notifier and the mappings to be re- negotiated.
Importers requiring an early hint as to what the actual mapping would look like would need to be prepared to re-consider at map-time, with a reasonable assumption that the reconsideration and re-mapping would happen seldom enough.
The idea was the supported mapping types are fixed properties of the importer and exporter.
Like iommufd is not suddenly going to start understanding ualink after an invalidation.
Further, I had the (perhaps naive) expectation that the interconnect option was not something that would change during mapping. I did not think you would loose your PCIe option if ualink becomes available, or vice versa. Though on deeper reflection I suppose someone could build HW so you might loose your ualink option if the dmabuf is moved to system memory while PCIe would still work.
True, but there are other complications. Let's say at map time there is another importer that don't support anything but system memory dma- bufs. Then both ualink and PCIe p2p could not be used, and we'd fall back to system. Or a map() move to VRAM from system for a fast interconnect failed for some reason. (Somebody might have pinned it in system).
But agreed it sounds to me like the interconnect capabilities should be fixed at attach time.
Making the decision right before map seems like a reasonable enhancement, but I can no longer remember off hand if that would cause complications :)
It was that mellanox ATS thing, right? That's the only case I heard of that needed the mapping type decision (or at least a strong hint) early.
/Thomas
Jason
On 10/19/26 15:47, Thomas Hellström wrote:
On Mon, 2026-10-19 at 10:30 -0300, Jason Gunthorpe wrote:
On Mon, Oct 19, 2026 at 02:43:29PM +0200, Thomas Hellström wrote:
On Mon, 2026-10-19 at 10:35 +0200, Christian König wrote:
On 10/16/26 22:26, Karl Mehltretter wrote:
On Tue, Feb 17, 2026 at 08:11:34PM -0400, Jason Gunthorpe wrote:
- match_args.attach = attach;
- if (dmabuf->ops->match_mapping) {
ret = dmabuf->ops-> match_mapping(&match_args);
- }
attach->dmabuf = dmabuf; attach->importer_ops = importer_ops; attach->importer_priv = importer_priv;
I may misunderstand the intended contract. match_mapping() runs while the attachment is still zeroed and without dmabuf->resv held.
Patch 11's amdgpu callback needs importer_ops and importer_priv, so its XGMI match cannot work here. Patch 12's VFIO callback asserts dmabuf-
resv and
reads priv->vdev, which cleanup changes under that lock.
Should the core initialize the attachment and hold dmabuf->resv for match_mapping(), or should callbacks avoid attachment and lock- protected state?
Right. This completely misplaced.
The matching can't be done at attachment time, but rather needs to be done before creating a mapping while holding the resv lock so that the matching decision can't change.
So question is what exactly should the contract be? Looking at the pcie_p2p negotiation today it looks somewhat like.
- Importer and exporter agree at attach time what mappings are
allowed pending placement conditions and other attachers.
Not quite. They check if the P2P is in theory possible, but they don't make any decisions on the final placement.
- At map-time the mapping is negotiated taking all constraints
into account. 3) The negotiated mapping is valid until an invalidate_mappings() callback is received.
Following this would allow a re-implementation of pcie_p2p as a mapping type and would allow a controlled link-severing to send an invalidate_mappings() notifier and the mappings to be re- negotiated.
Importers requiring an early hint as to what the actual mapping would look like would need to be prepared to re-consider at map-time, with a reasonable assumption that the reconsideration and re-mapping would happen seldom enough.
The idea was the supported mapping types are fixed properties of the importer and exporter.
That doesn't work quite like this.
Like iommufd is not suddenly going to start understanding ualink after an invalidation.
Further, I had the (perhaps naive) expectation that the interconnect option was not something that would change during mapping. I did not think you would loose your PCIe option if ualink becomes available, or vice versa. Though on deeper reflection I suppose someone could build HW so you might loose your ualink option if the dmabuf is moved to system memory while PCIe would still work.
True, but there are other complications. Let's say at map time there is another importer that don't support anything but system memory dma- bufs. Then both ualink and PCIe p2p could not be used, and we'd fall back to system. Or a map() move to VRAM from system for a fast interconnect failed for some reason. (Somebody might have pinned it in system).
Yes, exactly that.
But agreed it sounds to me like the interconnect capabilities should be fixed at attach time.
We can check preconditions, e.g. if PCI P2P would work with this root complex/isolation setup etc...
Making the decision right before map seems like a reasonable enhancement, but I can no longer remember off hand if that would cause complications :)
It was that mellanox ATS thing, right? That's the only case I heard of that needed the mapping type decision (or at least a strong hint) early.
Yeah I agree that this is the only one we have heard of, but that doesn't mean we shouldn't at least try to support it somehow.
I mean what we can do is to narrow down the possible placements on attach so that mellanox ATS can at least rule out that P2P is possible early and eventually stop wasting time setting that up.
It can still be that P2P or whatever interconnect fails later on, we have very nice "the fast interconnect has a failure use the slow backup" use cases as well, so importers really won't get away without handling that even if it's slow.
Regards, Christian.
/Thomas
Jason
On Mon, Oct 19, 2026 at 03:47:55PM +0200, Thomas Hellström wrote:
Making the decision right before map seems like a reasonable
enhancement, but I can no longer remember off hand if that would cause complications :)
It was that mellanox ATS thing, right? That's the only case I heard of that needed the mapping type decision (or at least a strong hint) early.
No, it would work after the fact just mlx5 forms the attachment and the map approximately at the same time in most important cases.
Also after LPC I've also realized that UIO shares alot of similarity with ATS, except getting UIO correct on both sides is functionally required not just a performance optimization :\
Jason
On 10/19/26 15:30, Jason Gunthorpe wrote:
On Mon, Oct 19, 2026 at 02:43:29PM +0200, Thomas Hellström wrote:
On Mon, 2026-10-19 at 10:35 +0200, Christian König wrote:
On 10/16/26 22:26, Karl Mehltretter wrote:
On Tue, Feb 17, 2026 at 08:11:34PM -0400, Jason Gunthorpe wrote:
- match_args.attach = attach;
- if (dmabuf->ops->match_mapping) {
ret = dmabuf->ops->match_mapping(&match_args);- }
attach->dmabuf = dmabuf; attach->importer_ops = importer_ops; attach->importer_priv = importer_priv;
I may misunderstand the intended contract. match_mapping() runs while the attachment is still zeroed and without dmabuf->resv held.
Patch 11's amdgpu callback needs importer_ops and importer_priv, so its XGMI match cannot work here. Patch 12's VFIO callback asserts dmabuf-
resv and
reads priv->vdev, which cleanup changes under that lock.
Should the core initialize the attachment and hold dmabuf->resv for match_mapping(), or should callbacks avoid attachment and lock- protected state?
Right. This completely misplaced.
The matching can't be done at attachment time, but rather needs to be done before creating a mapping while holding the resv lock so that the matching decision can't change.
So question is what exactly should the contract be? Looking at the pcie_p2p negotiation today it looks somewhat like.
- Importer and exporter agree at attach time what mappings are allowed
pending placement conditions and other attachers. 2) At map-time the mapping is negotiated taking all constraints into account. 3) The negotiated mapping is valid until an invalidate_mappings() callback is received.
Following this would allow a re-implementation of pcie_p2p as a mapping type and would allow a controlled link-severing to send an invalidate_mappings() notifier and the mappings to be re-negotiated.
Importers requiring an early hint as to what the actual mapping would look like would need to be prepared to re-consider at map-time, with a reasonable assumption that the reconsideration and re-mapping would happen seldom enough.
The idea was the supported mapping types are fixed properties of the importer and exporter.
Like iommufd is not suddenly going to start understanding ualink after an invalidation.
Yeah but I think we can sort things out early on.
E.g. even if both devices can do PCI P2P we can't use it when they are not under the same root complex and there is no routing between root complexes.
Or both devices have an XGMI connection with other devices but are not physically connected to each other, etc...
That is something which doesn't change unless you physically change the system.
Further, I had the (perhaps naive) expectation that the interconnect option was not something that would change during mapping. I did not think you would loose your PCIe option if ualink becomes available, or vice versa. Though on deeper reflection I suppose someone could build HW so you might loose your ualink option if the dmabuf is moved to system memory while PCIe would still work.
Yeah, that is definitely not correct.
We have the clear requirement here that when one interconnect physically fails that we can still do things like let the work reach the next checkpoint, then it is taken of the HW, and the HW suspended for repair, then resumed and then work can continue....
Similarly we have on the graphics side the requirement that physical placement can be moved to system memory because of some other device attaching, pinning for scanout or something suddenly requires the backing store to be contiguously allocated etc,,,
So there are vast requirements that a buffer changes location when the mapping changes. I mean it is kind of pointless to have a feature like ODP to suspend your traffic if the underlying physical location of the memory can't change.
Christian.
Making the decision right before map seems like a reasonable enhancement, but I can no longer remember off hand if that would cause complications :)
Jason
On Tue, Oct 20, 2026 at 09:01:33AM +0200, Christian König wrote:
Like iommufd is not suddenly going to start understanding ualink after an invalidation.
Yeah but I think we can sort things out early on.
Yeah, if ther is some value thing can be excluded right from the start.
Further, I had the (perhaps naive) expectation that the interconnect option was not something that would change during mapping. I did not think you would loose your PCIe option if ualink becomes available, or vice versa. Though on deeper reflection I suppose someone could build HW so you might loose your ualink option if the dmabuf is moved to system memory while PCIe would still work.
Yeah, that is definitely not correct.
We have the clear requirement here that when one interconnect physically fails that we can still do things like let the work reach the next checkpoint, then it is taken of the HW, and the HW suspended for repair, then resumed and then work can continue....
Similarly we have on the graphics side the requirement that physical placement can be moved to system memory because of some other device attaching, pinning for scanout or something suddenly requires the backing store to be contiguously allocated etc,,,
So there are vast requirements that a buffer changes location when the mapping changes. I mean it is kind of pointless to have a feature like ODP to suspend your traffic if the underlying physical location of the memory can't change.
At least here, I am not aware of anyone actually using the ODP capabilities to move and migrate memory. It seems by far and away the most common use case is permanently pin things.
It seems like in these very complicated chips the placement of the memory is a critical decision, if it moves around the performance is so destroyed you may as well not bother.
I know this is not everyone and every use case, but it has been a bit eye opening we built all this complex ODP stuf and I struggle to justify it with an actual real user.
But yes, we should try to build it as you describe.
Thanks, Jason
linaro-mm-sig@lists.linaro.org
-
Christian König -
Jason Gunthorpe -
Thomas Hellström