On Mon, Apr 14, 2025 at 10:24:55AM +0200, Thomas Petazzoni wrote:
> What this patch series is about is to add new user-space interface to
> extend the existing UIO subsystem.
Which as I explained to you is fundamentally broken and unsafe. If you
need to do DMA from userspae you need to use vfio/iommufd.
> I am not sure how this can work in our use-case. We have a very simple
> set of IP blocks implemented in a FPGA, some of those IP blocks are
> able to perform DMA operations. The register of those IP blocks are
> mapped into a user-space application using the existing, accepted
> upstream, UIO subsystem. Some of those registers allow to program DMA
> transfers. So far, we can do all what we need, except program those DMA
> transfers. Lots of people are having the same issue, and zillions of
> ugly out-of-tree solutions flourish all over, and we're trying to see
> if we can constructively find a solution that would be acceptable
> upstream to resolve this use-case. Our platform is an old PowerPC with
> no IOMMU.
Then your driver design can't work and you need to replace it with a
proper in-kernel driver.
Am 14.04.25 um 10:24 schrieb Thomas Petazzoni:
> Hello Christoph,
>
> On Sun, 13 Apr 2025 22:55:02 -0700
> Christoph Hellwig <hch(a)infradead.org> wrote:
>
>> On Thu, Apr 10, 2025 at 04:53:17PM +0200, Bastien Curutchet wrote:
>>> Hi all,
>>>
>>> Many UIO users performing DMA from their UIO device need to access the
>>> DMA addresses of the allocated buffers. There are out-of-tree drivers
>>> that allow to do it but nothing in the mainline.
>> In which case it does not matter at all for mainline.
> It is impressive how "out-of-tree" triggers kernel maintainers, and
> then end up not even reading what the patch series is all about (but I
> forgive you, because it triggers me in the same way).
>
> This patch series is NOT about adding new kernel APIs meant to be used
> by out-of-tree drivers, which of course would be bad, and we would have
> never proposed.
>
> What this patch series is about is to add new user-space interface to
> extend the existing UIO subsystem. What my colleague Bastien was
> mentioning about out-of-tree drivers is that there are a LOT of
> out-of-tree drivers extending UIO to allow user-space applications to
> do DMA, and our proposal is that considering how many people need that
> and implement ugly out-of-tree drivers to solve this issue, we suggest
> the mainline kernel should have a built in solution.
>
> Please re-read again, and realize that we are NOT adding new kernel
> APIs for out-of-tree drivers.
>
>> FYI the proper and safe way to do DMA from userspace is to use
>> vfio/iommufd.
> I am not sure how this can work in our use-case. We have a very simple
> set of IP blocks implemented in a FPGA, some of those IP blocks are
> able to perform DMA operations. The register of those IP blocks are
> mapped into a user-space application using the existing, accepted
> upstream, UIO subsystem. Some of those registers allow to program DMA
> transfers. So far, we can do all what we need, except program those DMA
> transfers. Lots of people are having the same issue, and zillions of
> ugly out-of-tree solutions flourish all over, and we're trying to see
> if we can constructively find a solution that would be acceptable
> upstream to resolve this use-case. Our platform is an old PowerPC with
> no IOMMU.
Maybe I should try to better explain the concern here. The question is "Where is the source code of your FPGA driver?".
I mean that you are trying to replace the out-of-tree solution is rather welcomed, but the out-of-tree solutions are out-of-tree because they don't fit with requirements to be added to the core Linux tree.
And one of those requirements is that you need to provide the source code of the userspace user of this interface, in this case here that is your FPGA driver. An MIT/X11 license is usually sufficient, GPL is of course seen as better and it must not be a toy application, but rather the real thing.
And that is what people usually don't want and that's why no in-tree solution exists for this.
Regards,
Christian.
>
> Note that the safety argument doesn't hold: UIO already allows to map
> registers into user-space applications, which can already program DMA
> transfers to arbitrary locations in memory. The safety comes from the
> fact that such UIO devices are associated with permissions that the
> system administrator controls, to decide which applications are safe
> enough to be granted access to those UIO devices. Therefore, providing
> access through UIO of the physical address of well-defined DMA buffers
> properly allocated through a sane API is not reducing the safety of
> anything compared to what UIO already allows today.
>
> Best regards,
>
> Thomas
Am 14.04.25 um 10:17 schrieb Thomas Petazzoni:
> Hello Christian,
>
> On Fri, 11 Apr 2025 14:41:56 +0200
> Christian König <christian.koenig(a)amd.com> wrote:
>
>> But anyway please note that when you want to create new UAPI you need
>> to provide an open source user of it. E.g. link to a repository or
>> something similar in the covert letter should do it.
> Could you clarify what is the "reference" user-space user of UIO that
> exists today?
I honestly don't know. I have never looked into UIO before this patch set arrived.
What I mean is that the general rule to justify adding UAPI to the Linux kernel is that you need to have an open source user of that UAPI.
In other words for UIO this means that you *can't* do things like exposing some kernel functionality like DMA-buf through UIO and then write a binary only driver on top of it.
Regards,
Christian.
>
> Thomas
On Thu, Apr 10, 2025 at 04:53:17PM +0200, Bastien Curutchet wrote:
> Hi all,
>
> Many UIO users performing DMA from their UIO device need to access the
> DMA addresses of the allocated buffers. There are out-of-tree drivers
> that allow to do it but nothing in the mainline.
In which case it does not matter at all for mainline.
FYI the proper and safe way to do DMA from userspace is to use
vfio/iommufd.
Am 11.04.25 um 14:44 schrieb Philipp Stanner:
> On Fri, 2025-04-11 at 13:05 +0200, Christian König wrote:
>> Am 11.04.25 um 11:29 schrieb Philipp Stanner:
>>
>>> [SNIP]
>>>
>>> It could be, however, that at the same moment
>>> nouveau_fence_signal() is
>>> removing that entry, holding the appropriate lock.
>>>
>>> So we have a race. Again.
>>>
>>
>> Ah, yes of course. If signaled is called with or without the lock is
>> actually undetermined.
>>
>>
>>>
>>> You see, fixing things in Nouveau is difficult :)
>>> It gets more difficult if you want to clean it up "properly", so it
>>> conforms to rules such as those from dma_fence.
>>>
>>> I have now provided two fixes that both work, but you are not
>>> satisfied
>>> with from the dma_fence-maintainer's perspective. I understand
>>> that,
>>> but please also understand that it's actually not my primary task
>>> to
>>> work on Nouveau. I just have to fix this bug to move on with my
>>> scheduler work.
>>>
>>
>> Well I'm happy with whatever solution as long as it works, but as
>> far as I can see the approach with the callback simply doesn't.
>>
>> You just can't drop the fence reference for the list from the
>> callback.
>>
>>
>>>
>>> So if you have another idea, feel free to share it. But I'd like to
>>> know how we can go on here.
>>>
>>
>> Well the fence code actually works, doesn't it? The problem is
>> rather that setting the error throws a warning because it doesn't
>> expect signaled fences on the pending list.
>>
>> Maybe we should fix that instead.
> The fence code works as the author intended, but I would be happy if it
> were more explicitly documented.
>
> Regarding the WARN_ON: It occurs in dma_fence_set_error() because there
> is an attempt to set an error code on a signaled fence. I don't think
> that should be "fixed", it works as intended: You must not set an error
> code of a fence that was already signaled.
>
> The reason seems to be that once a fence is signaled, a third party
> might evaluate the error code.
Yeah, more or less correct. The idea is you can't declare an operation as having an error after the operation has already completed.
Because everyone will just wait for the completion and nobody checks the status again after that.
>
> But I think this wasn't wat you meant with "fix".
The idea was to avoid calling dma_fence_set_error() on already signaled fences. Something like this:
@@ -90,7 +90,7 @@ nouveau_fence_context_kill(struct nouveau_fence_chan *fctx, int error)
while (!list_empty(&fctx->pending)) {
fence = list_entry(fctx->pending.next, typeof(*fence), head);
- if (error)
+ if (error & !dma_fence_is_signaled_locked(&fence->base))
dma_fence_set_error(&fence->base, error);
if (nouveau_fence_signal(fence))
That would also improve the handling quite a bit since we now don't set errors on fences which are already completed even if we haven't realized that they are already completed yet.
> In any case, there must not be signaled fences in nouveau's pending-
> list. They must be removed immediately once they signal, and this must
> not race.
Why actually? As far as I can see the pending list is not for the unsignaled fences, but rather the pending interrupt processing.
So having signaled fences on the pending list is perfectly possible.
Regards,
Christian.
>
>>
>>
>>>
>>> I'm running out of ideas. What I'm wondering if we couldn't just
>>> remove
>>> performance hacky fastpath functions such as
>>> nouveau_fence_is_signaled() completely. It seems redundant to me.
>>>
>>
>> That would work for me as well.
> I'll test this approach. Seems a bit like the nuclear approach, but if
> it works we'd at least clean up a lot of this mess.
>
>
> P.
>
>
>>
>>
>>>
>>>
>>> Or we might add locking to it, but IDK what was achieved with RCU
>>> here.
>>> In any case it's definitely bad that Nouveau has so many redundant
>>> and
>>> half-redundant mechanisms.
>>>
>>
>> Yeah, agree messing with the locks even more won't help us here.
>>
>> Regards,
>> Christian.
>>
>>
>>>
>>>
>>>
>>> P.
>>>
>>>
>>>>
>>>>
>>>> P.
>>>>
>>>>
>>>>>
>>>>> Regards,
>>>>> Christian.
>>>>>
>>>>>
>>>>>>
>>>>>> P.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Christian.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Replace the call to dma_fence_is_signaled() with
>>>>>>>> nouveau_fence_base_is_signaled().
>>>>>>>>
>>>>>>>> Cc: <stable(a)vger.kernel.org> # 4.10+, precise commit not
>>>>>>>> to
>>>>>>>> be
>>>>>>>> determined
>>>>>>>> Signed-off-by: Philipp Stanner <phasta(a)kernel.org>
>>>>>>>> ---
>>>>>>>> drivers/gpu/drm/nouveau/nouveau_fence.c | 2 +-
>>>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>>>
>>>>>>>> diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> index 7cc84472cece..33535987d8ed 100644
>>>>>>>> --- a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> @@ -274,7 +274,7 @@ nouveau_fence_done(struct
>>>>>>>> nouveau_fence
>>>>>>>> *fence)
>>>>>>>> nvif_event_block(&fctx->event);
>>>>>>>> spin_unlock_irqrestore(&fctx->lock,
>>>>>>>> flags);
>>>>>>>> }
>>>>>>>> - return dma_fence_is_signaled(&fence->base);
>>>>>>>> + return test_bit(DMA_FENCE_FLAG_SIGNALED_BIT,
>>>>>>>> &fence-
>>>>>>>>
>>>>>>>>>
>>>>>>>>> base.flags);
>>>>>>>>>
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> static long
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
Hi Bastien,
Am 11.04.25 um 10:14 schrieb Bastien Curutchet:
> Hi Christian,
>
> On 4/11/25 9:30 AM, Christian König wrote:
>> Hi Thomas,
>>
>> Am 10.04.25 um 21:43 schrieb Thomas Petazzoni:
>>> Hello Christian,
>>>
>>> Thanks for your feedback!
>>>
>>> On Thu, 10 Apr 2025 18:29:12 +0200
>>> Christian König<christian.koenig(a)amd.com> wrote:
>>>
>>>>> Many UIO users performing DMA from their UIO device need to access the
>>>>> DMA addresses of the allocated buffers. There are out-of-tree drivers
>>>>> that allow to do it but nothing in the mainline.
>>>> Well that basically disqualifies this patch set in the first paragraph.
>>>>
>>>> To justify some kernel change we always need an in kernel user of the
>>>> interface, since this is purely for out-of-tree drivers this is a
>>>> no-go to begin with.
>>> I'm not sure to understand your comment here. This patch series is
>>> about extending the UIO interface... which is a user-space interface.
>>> So obviously it has no "in-kernel user" because it's meant to be used
>>> from user-space. Could you clarify what you meant here?
>>
>> Bastien wrote about "out-of-tree drivers" which is something the upstream kernel explicitly does not support.
>>
>
> Sorry maybe it wasn't clear, but what I meant is that the goal of this series is to replace 'out-of-tree drivers' with something upstream.
Ah! Yeah that wasn't really clear from the description.
But anyway please note that when you want to create new UAPI you need to provide an open source user of it. E.g. link to a repository or something similar in the covert letter should do it.
>> Well why do you then want to use DMA-buf in the first place? As far as I know what you describe can perfectly be done with the normal UIO memory management interfaces.
>>
>> DMA-buf is only interesting when you actually want to share something in between devices or between applications.
>>
>
> I wanted to use DMA-buf because it allows to dynamically allocate/release coherent buffers from userspace. UIO doesn't provide such interface.
> I'm aware that exposing DMA addresses to userspace isn't a good practice. That's why this series create a new heap specific to UIO that will be the only one implementing the new ioctl.
I don't know the UIO interfaces that well, but that is pretty clearly an abuse of DMA-buf and won't fly at all.
If you want coherent memory for your device you should use dma_alloc_coherent() for that.
Regards,
Christian.
>
>
> Best regards,
> Bastien
>
>
>
Am 11.04.25 um 11:29 schrieb Philipp Stanner:
> [SNIP]
> It could be, however, that at the same moment nouveau_fence_signal() is
> removing that entry, holding the appropriate lock.
>
> So we have a race. Again.
Ah, yes of course. If signaled is called with or without the lock is actually undetermined.
> You see, fixing things in Nouveau is difficult :)
> It gets more difficult if you want to clean it up "properly", so it
> conforms to rules such as those from dma_fence.
>
> I have now provided two fixes that both work, but you are not satisfied
> with from the dma_fence-maintainer's perspective. I understand that,
> but please also understand that it's actually not my primary task to
> work on Nouveau. I just have to fix this bug to move on with my
> scheduler work.
Well I'm happy with whatever solution as long as it works, but as far as I can see the approach with the callback simply doesn't.
You just can't drop the fence reference for the list from the callback.
> So if you have another idea, feel free to share it. But I'd like to
> know how we can go on here.
Well the fence code actually works, doesn't it? The problem is rather that setting the error throws a warning because it doesn't expect signaled fences on the pending list.
Maybe we should fix that instead.
> I'm running out of ideas. What I'm wondering if we couldn't just remove
> performance hacky fastpath functions such as
> nouveau_fence_is_signaled() completely. It seems redundant to me.
That would work for me as well.
>
> Or we might add locking to it, but IDK what was achieved with RCU here.
> In any case it's definitely bad that Nouveau has so many redundant and
> half-redundant mechanisms.
Yeah, agree messing with the locks even more won't help us here.
Regards,
Christian.
>
>
> P.
>
>>
>> P.
>>
>>> Regards,
>>> Christian.
>>>
>>>> P.
>>>>
>>>>
>>>>
>>>>> Regards,
>>>>> Christian.
>>>>>
>>>>>> Replace the call to dma_fence_is_signaled() with
>>>>>> nouveau_fence_base_is_signaled().
>>>>>>
>>>>>> Cc: <stable(a)vger.kernel.org> # 4.10+, precise commit not to
>>>>>> be
>>>>>> determined
>>>>>> Signed-off-by: Philipp Stanner <phasta(a)kernel.org>
>>>>>> ---
>>>>>> drivers/gpu/drm/nouveau/nouveau_fence.c | 2 +-
>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>> b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>> index 7cc84472cece..33535987d8ed 100644
>>>>>> --- a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>> +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>> @@ -274,7 +274,7 @@ nouveau_fence_done(struct nouveau_fence
>>>>>> *fence)
>>>>>> nvif_event_block(&fctx->event);
>>>>>> spin_unlock_irqrestore(&fctx->lock, flags);
>>>>>> }
>>>>>> - return dma_fence_is_signaled(&fence->base);
>>>>>> + return test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence-
>>>>>>> base.flags);
>>>>>> }
>>>>>>
>>>>>> static long
Hi Thomas,
Am 10.04.25 um 21:43 schrieb Thomas Petazzoni:
> Hello Christian,
>
> Thanks for your feedback!
>
> On Thu, 10 Apr 2025 18:29:12 +0200
> Christian König <christian.koenig(a)amd.com> wrote:
>
>>> Many UIO users performing DMA from their UIO device need to access the
>>> DMA addresses of the allocated buffers. There are out-of-tree drivers
>>> that allow to do it but nothing in the mainline.
>> Well that basically disqualifies this patch set in the first paragraph.
>>
>> To justify some kernel change we always need an in kernel user of the
>> interface, since this is purely for out-of-tree drivers this is a
>> no-go to begin with.
> I'm not sure to understand your comment here. This patch series is
> about extending the UIO interface... which is a user-space interface.
> So obviously it has no "in-kernel user" because it's meant to be used
> from user-space. Could you clarify what you meant here?
Bastien wrote about "out-of-tree drivers" which is something the upstream kernel explicitly does not support.
When you make that UIO API and have an open source userspace driver then that is probably a good justification to do this.
What the kernel community tries to prevent here is that people start using the UAPI to write closed source drivers in userspace.
>> What you could potentially do is to create an UIO driver which
>> imports DMA-bufs, pins them and then provide the DMA addresses to
>> userspace.
>>
>> But please be aware that DMA-fences are fundamentally incompatible
>> with UIO. So you won't be able to do any form of synchronization
>> which probably makes the implementation pretty limited.
> Could you clarify why DMA-fences would be needed here, and for what
> synchronization?
In general DMA-buf is an interface which enables you do share device specific buffers between different drivers as well as between userspace processes.
For this to work with for example cameras, GPUs or RDMA NICs you need to some kind of synchronization primitive, e.g. device A can only starts it's DMA transaction when device B has completed his.
The problem is that this synchronization approach is fundamentally incompatible with UIO. See here for more details: https://www.kernel.org/doc/html/latest/driver-api/dma-buf.html#indefinite-d…
> The DMA buffers allocated here are DMA coherent buffers. So the
> user-space application that uses UIO will allocate such buffers once at
> application start, retrieve their DMA address, and then program DMA
> transfers as it sees fit using the memory-mapped registers accessible
> through UIO. I'm not sure which synchronization you are referring to.
> We are not "chaining" DMA transfers, with for example a camera
> interface feeding its DMA buffers to an ISP or something like that. The
> typical use case here is some IP block in an FPGA that does DMA
> transfers to transfer data to/from some sort of specialized I/O
> interface. We get an interrupt when the transfer is done, and we can
> re-use the buffer for the next transfer.
Well why do you then want to use DMA-buf in the first place? As far as I know what you describe can perfectly be done with the normal UIO memory management interfaces.
DMA-buf is only interesting when you actually want to share something in between devices or between applications.
Regards,
Christian.
> If you could clarify here as well, it would definitely help in
> understanding the shortcomings/limitations.
>
> Thanks a lot!
>
> Thomas Petazzoni
Am 10.04.25 um 16:53 schrieb Bastien Curutchet:
> Hi all,
>
> Many UIO users performing DMA from their UIO device need to access the
> DMA addresses of the allocated buffers. There are out-of-tree drivers
> that allow to do it but nothing in the mainline.
Well that basically disqualifies this patch set in the first paragraph.
To justify some kernel change we always need an in kernel user of the interface, since this is purely for out-of-tree drivers this is a no-go to begin with.
> I know DMA shouldn't be handled by userspace but, IMHO, since UIO
> drivers exist, it would be better if they offered a way of doing this.
Leaking DMA addresses to userspace is usually seen as quite some security hole, but on the other hand with UIO you don't have much other choice.
> This patch series use the dma-heap framework which already allows
> userspace to allocate DMA buffers. I tried to avoid 'polluting' the
> existing heaps to prevent inappropriate uses of this new feature by
> introducing a new UIO heap, which is the only one implementing this
> behavior.
Yeah, that won't fly at all.
What you could potentially do is to create an UIO driver which imports DMA-bufs, pins them and then provide the DMA addresses to userspace.
But please be aware that DMA-fences are fundamentally incompatible with UIO. So you won't be able to do any form of synchronization which probably makes the implementation pretty limited.
Regards,
Christian.
>
> PATCH 1 allows the creation of heaps that don't implement map/unmap_buf
> operations as UIO heap doesn't use them.
> PATCH 2 adds the DMA_BUF_IOCTL_GET_DMA_ADDR which transmits the DMA
> addresses to userspace.
> PATCH 3 implements the UIO heap.
>
> It has been tested with the uio_pci_generic driver on a PowerPC.
>
> Signed-off-by: Bastien Curutchet <bastien.curutchet(a)bootlin.com>
> ---
> Bastien Curutchet (3):
> dma-buf: Allow heap that doesn't provide map_buf/unmap_buf
> dma-buf: Add DMA_BUF_IOCTL_GET_DMA_ADDR
> uio: Add UIO_DMABUF_HEAP
>
> drivers/dma-buf/dma-buf.c | 29 +++++++++--
> drivers/uio/Kconfig | 9 ++++
> drivers/uio/Makefile | 1 +
> drivers/uio/uio.c | 4 ++
> drivers/uio/uio_heap.c | 120 +++++++++++++++++++++++++++++++++++++++++++
> include/linux/dma-buf.h | 1 +
> include/linux/uio_driver.h | 2 +
> include/uapi/linux/dma-buf.h | 1 +
> 8 files changed, 164 insertions(+), 3 deletions(-)
> ---
> base-commit: 5f13fa25acaa4f586aaed12efcf7436e004eeaf2
> change-id: 20250408-uio-dma-9b011e9e7f0b
>
> Best regards,
Am 10.04.25 um 15:09 schrieb Philipp Stanner:
> On Thu, 2025-04-10 at 14:58 +0200, Christian König wrote:
>> Am 10.04.25 um 11:24 schrieb Philipp Stanner:
>>> Nouveau currently relies on the assumption that dma_fences will
>>> only
>>> ever get signaled through nouveau_fence_signal(), which takes care
>>> of
>>> removing a signaled fence from the list nouveau_fence_chan.pending.
>>>
>>> This self-imposed rule is violated in nouveau_fence_done(), where
>>> dma_fence_is_signaled() (somewhat surprisingly, considering its
>>> name)
>>> can signal the fence without removing it from the list. This
>>> enables
>>> accesses to already signaled fences through the list, which is a
>>> bug.
>>>
>>> In particular, it can race with nouveau_fence_context_kill(), which
>>> would then attempt to set an error code on an already signaled
>>> fence,
>>> which is illegal.
>>>
>>> In nouveau_fence_done(), the call to nouveau_fence_update() already
>>> ensures to signal all ready fences. Thus, the signaling potentially
>>> performed by dma_fence_is_signaled() is actually not necessary.
>> Ah, I now got what you are trying to do here! But that won't help.
>>
>> The problem is it is perfectly valid for somebody external (e.g.
>> other driver, TTM etc...) to call dma_fence_is_signaled() on a
>> nouveau fence.
>>
>> This will then in turn still signal the fence and leave it on the
>> pending list and creating the problem you have.
> Good to hear – precisely that then is the use case for a dma_fence
> callback! ^_^ It guarantees that, no matter who signals a fence, no
> matter at what place, a certain action will always be performed.
>
> I can't think of any other mechanism which could guarantee that a
> signaled fence immediately gets removed from nouveau's pending list,
> other than the callbacks.
>
> But seriously, I don't think that anyone does this currently, nor do I
> think that anyone could get away with doing it without the entire
> computer burning down.
Yeah, I don't think that this is possible at the moment.
When you do stuff like that from the provider side you will always run into lifetime issues because in the signaling from interrupt case you then drop the last reference before the signaling is completed.
How about the attached (not even compile tested) patch? I think it should fix the issue.
Regards,
Christian.
>
> P.
>
>
>
>> Regards,
>> Christian.
>>
>>> Replace the call to dma_fence_is_signaled() with
>>> nouveau_fence_base_is_signaled().
>>>
>>> Cc: <stable(a)vger.kernel.org> # 4.10+, precise commit not to be
>>> determined
>>> Signed-off-by: Philipp Stanner <phasta(a)kernel.org>
>>> ---
>>> drivers/gpu/drm/nouveau/nouveau_fence.c | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>> b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>> index 7cc84472cece..33535987d8ed 100644
>>> --- a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>> +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>> @@ -274,7 +274,7 @@ nouveau_fence_done(struct nouveau_fence *fence)
>>> nvif_event_block(&fctx->event);
>>> spin_unlock_irqrestore(&fctx->lock, flags);
>>> }
>>> - return dma_fence_is_signaled(&fence->base);
>>> + return test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence-
>>>> base.flags);
>>> }
>>>
>>> static long
Am 10.04.25 um 11:24 schrieb Philipp Stanner:
> Nouveau currently relies on the assumption that dma_fences will only
> ever get signaled through nouveau_fence_signal(), which takes care of
> removing a signaled fence from the list nouveau_fence_chan.pending.
>
> This self-imposed rule is violated in nouveau_fence_done(), where
> dma_fence_is_signaled() (somewhat surprisingly, considering its name)
> can signal the fence without removing it from the list. This enables
> accesses to already signaled fences through the list, which is a bug.
>
> In particular, it can race with nouveau_fence_context_kill(), which
> would then attempt to set an error code on an already signaled fence,
> which is illegal.
>
> In nouveau_fence_done(), the call to nouveau_fence_update() already
> ensures to signal all ready fences. Thus, the signaling potentially
> performed by dma_fence_is_signaled() is actually not necessary.
Ah, I now got what you are trying to do here! But that won't help.
The problem is it is perfectly valid for somebody external (e.g. other driver, TTM etc...) to call dma_fence_is_signaled() on a nouveau fence.
This will then in turn still signal the fence and leave it on the pending list and creating the problem you have.
Regards,
Christian.
>
> Replace the call to dma_fence_is_signaled() with
> nouveau_fence_base_is_signaled().
>
> Cc: <stable(a)vger.kernel.org> # 4.10+, precise commit not to be determined
> Signed-off-by: Philipp Stanner <phasta(a)kernel.org>
> ---
> drivers/gpu/drm/nouveau/nouveau_fence.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c b/drivers/gpu/drm/nouveau/nouveau_fence.c
> index 7cc84472cece..33535987d8ed 100644
> --- a/drivers/gpu/drm/nouveau/nouveau_fence.c
> +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c
> @@ -274,7 +274,7 @@ nouveau_fence_done(struct nouveau_fence *fence)
> nvif_event_block(&fctx->event);
> spin_unlock_irqrestore(&fctx->lock, flags);
> }
> - return dma_fence_is_signaled(&fence->base);
> + return test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->base.flags);
> }
>
> static long
Am 10.04.25 um 14:21 schrieb Danilo Krummrich:
> On Thu, Apr 10, 2025 at 02:13:34PM +0200, Christian König wrote:
>> Am 10.04.25 um 11:24 schrieb Philipp Stanner:
>>> Nouveau currently relies on the assumption that dma_fences will only
>>> ever get signaled through nouveau_fence_signal(), which takes care of
>>> removing a signaled fence from the list nouveau_fence_chan.pending.
>>>
>>> This self-imposed rule is violated in nouveau_fence_done(), where
>>> dma_fence_is_signaled() (somewhat surprisingly, considering its name)
>>> can signal the fence without removing it from the list. This enables
>>> accesses to already signaled fences through the list, which is a bug.
>>>
>>> In particular, it can race with nouveau_fence_context_kill(), which
>>> would then attempt to set an error code on an already signaled fence,
>>> which is illegal.
>>>
>>> In nouveau_fence_done(), the call to nouveau_fence_update() already
>>> ensures to signal all ready fences. Thus, the signaling potentially
>>> performed by dma_fence_is_signaled() is actually not necessary.
>>>
>>> Replace the call to dma_fence_is_signaled() with
>>> nouveau_fence_base_is_signaled().
>>>
>>> Cc: <stable(a)vger.kernel.org> # 4.10+, precise commit not to be determined
>>> Signed-off-by: Philipp Stanner <phasta(a)kernel.org>
>>> ---
>>> drivers/gpu/drm/nouveau/nouveau_fence.c | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>> index 7cc84472cece..33535987d8ed 100644
>>> --- a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>> +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>> @@ -274,7 +274,7 @@ nouveau_fence_done(struct nouveau_fence *fence)
>>> nvif_event_block(&fctx->event);
>>> spin_unlock_irqrestore(&fctx->lock, flags);
>>> }
>>> - return dma_fence_is_signaled(&fence->base);
>>> + return test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->base.flags);
>> See the code above that:
>>
>> if (fence->base.ops == &nouveau_fence_ops_legacy ||
>> fence->base.ops == &nouveau_fence_ops_uevent) {
> I think this check is a bit pointless given that fence is already a struct
> nouveau_fence. :)
Oh, good point. I totally missed that.
In this case that indeed doesn't make any sense at all.
(Unless somebody just blindly upcasted the structure, but I really hope that this isn't the case here).
Regards,
Christian.
Am 10.04.25 um 11:24 schrieb Philipp Stanner:
> nouveau_fence_done() contains an if-branch which checks for the
> existence of either of two fence backend ops. Those two are the only
> backend ops existing in Nouveau, however; and at least one backend ops
> must be in use for the entire driver to be able to work. The if branch
> is, therefore, surplus.
>
> Remove the if-branch.
What happens here is that nouveau checks if the fence comes from itself or some external source.
So when you remove that check you potentially illegally uses nouveau_fctx() on a non-nouveau fence.
Regards,
Christian.
>
> Signed-off-by: Philipp Stanner <phasta(a)kernel.org>
> ---
> drivers/gpu/drm/nouveau/nouveau_fence.c | 24 +++++++++++-------------
> 1 file changed, 11 insertions(+), 13 deletions(-)
>
> diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c b/drivers/gpu/drm/nouveau/nouveau_fence.c
> index 33535987d8ed..db6f4494405c 100644
> --- a/drivers/gpu/drm/nouveau/nouveau_fence.c
> +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c
> @@ -259,21 +259,19 @@ nouveau_fence_emit(struct nouveau_fence *fence)
> bool
> nouveau_fence_done(struct nouveau_fence *fence)
> {
> - if (fence->base.ops == &nouveau_fence_ops_legacy ||
> - fence->base.ops == &nouveau_fence_ops_uevent) {
> - struct nouveau_fence_chan *fctx = nouveau_fctx(fence);
> - struct nouveau_channel *chan;
> - unsigned long flags;
> + struct nouveau_fence_chan *fctx = nouveau_fctx(fence);
> + struct nouveau_channel *chan;
> + unsigned long flags;
>
> - if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->base.flags))
> - return true;
> + if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->base.flags))
> + return true;
> +
> + spin_lock_irqsave(&fctx->lock, flags);
> + chan = rcu_dereference_protected(fence->channel, lockdep_is_held(&fctx->lock));
> + if (chan && nouveau_fence_update(chan, fctx))
> + nvif_event_block(&fctx->event);
> + spin_unlock_irqrestore(&fctx->lock, flags);
>
> - spin_lock_irqsave(&fctx->lock, flags);
> - chan = rcu_dereference_protected(fence->channel, lockdep_is_held(&fctx->lock));
> - if (chan && nouveau_fence_update(chan, fctx))
> - nvif_event_block(&fctx->event);
> - spin_unlock_irqrestore(&fctx->lock, flags);
> - }
> return test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->base.flags);
> }
>
Am 10.04.25 um 11:24 schrieb Philipp Stanner:
> Nouveau currently relies on the assumption that dma_fences will only
> ever get signaled through nouveau_fence_signal(), which takes care of
> removing a signaled fence from the list nouveau_fence_chan.pending.
>
> This self-imposed rule is violated in nouveau_fence_done(), where
> dma_fence_is_signaled() (somewhat surprisingly, considering its name)
> can signal the fence without removing it from the list. This enables
> accesses to already signaled fences through the list, which is a bug.
>
> In particular, it can race with nouveau_fence_context_kill(), which
> would then attempt to set an error code on an already signaled fence,
> which is illegal.
>
> In nouveau_fence_done(), the call to nouveau_fence_update() already
> ensures to signal all ready fences. Thus, the signaling potentially
> performed by dma_fence_is_signaled() is actually not necessary.
>
> Replace the call to dma_fence_is_signaled() with
> nouveau_fence_base_is_signaled().
>
> Cc: <stable(a)vger.kernel.org> # 4.10+, precise commit not to be determined
> Signed-off-by: Philipp Stanner <phasta(a)kernel.org>
> ---
> drivers/gpu/drm/nouveau/nouveau_fence.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c b/drivers/gpu/drm/nouveau/nouveau_fence.c
> index 7cc84472cece..33535987d8ed 100644
> --- a/drivers/gpu/drm/nouveau/nouveau_fence.c
> +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c
> @@ -274,7 +274,7 @@ nouveau_fence_done(struct nouveau_fence *fence)
> nvif_event_block(&fctx->event);
> spin_unlock_irqrestore(&fctx->lock, flags);
> }
> - return dma_fence_is_signaled(&fence->base);
> + return test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->base.flags);
See the code above that:
if (fence->base.ops == &nouveau_fence_ops_legacy ||
fence->base.ops == &nouveau_fence_ops_uevent) {
....
Nouveau first tests if it's one of it's own fences, and if yes does some special handling. E.g. checking the fence status bits etc...
So this dma_fence_is_signaled() is for all non-nouveau fences and then not touching the internal flags is perfectly correct as far as I can see.
Regards,
Christian.
> }
>
> static long
Am 09.04.25 um 17:04 schrieb Philipp Stanner:
> On Wed, 2025-04-09 at 16:10 +0200, Christian König wrote:
>>> I only see improvement by making things more obvious.
>>>
>>> In any case, how would you call a wrapper that just does
>>> test_bit(IS_SIGNALED, …) ?
>> Broken, that was very intentionally removed quite shortly after we
>> created the framework.
>>
>> We have a few cases were implementations do check that for their
>> fences, but consumers should never be allowed to touch such
>> internals.
> There is theory and there is practice. In practice, those internals are
> being used by Nouveau, i915, Xe, vmgfx and radeon.
What do you mean? I only skimmed over the use cases, but as far as I can see those are all valid.
You can test the flag if you know what the fence means to you, that is not a problem at all.
> So it seems that we failed quite a bit at communicating clearly how the
> interface should be used.
>
> And, to repeat myself, with both name and docu of that function, I
> think it is very easy to misunderstand what it's doing. You say that it
> shouldn't matter – and maybe that's true, in theory. In practice, it
> does matter. In practice, APIs get misused and have side-effects. And
> making that harder is desirable.
That sounds like I didn't used the right wording.
It *must* not matter to the consumer. See the purpose of the DMA-fence framework is to make it irrelevant for the consumer how the provider has implemented it's fences.
This means that things like if polling or interrupt driven signaling is used, 32bit vs 64bit seq numbers, etc... should all be hidden by the framework from the consumer of the fences.
BTW I'm actually not sure if nouveau has a bug here. As far as I can see nouveau_fence_signal() will be called later eventually and do the necessary cleanup.
But on the other hand it wouldn't surprise me if nouveau has a bug with that. The driver has been basically only barely maintained for quite a while.
> In any case, I might have to add another such call to Nouveau, because
> the solution preferred by you over the callback causes another race.
> Certainly one could solve this in a clean way, but someone has to do
> the work, and we're talking about more than a few hours here.
Well this is not my preferred solution, it's just the technical correct solution as far as I can see.
> In any case, be so kind and look at patch 2 and tell me there if you're
> at least OK with making the documentation more detailed.
As far as I can see that is clearly the wrong place to document that stuff.
Regards,
Christian.
>
> P.