Am 14.04.25 um 10:24 schrieb Thomas Petazzoni:
> Hello Christoph,
>
> On Sun, 13 Apr 2025 22:55:02 -0700
> Christoph Hellwig <hch(a)infradead.org> wrote:
>
>> On Thu, Apr 10, 2025 at 04:53:17PM +0200, Bastien Curutchet wrote:
>>> Hi all,
>>>
>>> Many UIO users performing DMA from their UIO device need to access the
>>> DMA addresses of the allocated buffers. There are out-of-tree drivers
>>> that allow to do it but nothing in the mainline.
>> In which case it does not matter at all for mainline.
> It is impressive how "out-of-tree" triggers kernel maintainers, and
> then end up not even reading what the patch series is all about (but I
> forgive you, because it triggers me in the same way).
>
> This patch series is NOT about adding new kernel APIs meant to be used
> by out-of-tree drivers, which of course would be bad, and we would have
> never proposed.
>
> What this patch series is about is to add new user-space interface to
> extend the existing UIO subsystem. What my colleague Bastien was
> mentioning about out-of-tree drivers is that there are a LOT of
> out-of-tree drivers extending UIO to allow user-space applications to
> do DMA, and our proposal is that considering how many people need that
> and implement ugly out-of-tree drivers to solve this issue, we suggest
> the mainline kernel should have a built in solution.
>
> Please re-read again, and realize that we are NOT adding new kernel
> APIs for out-of-tree drivers.
>
>> FYI the proper and safe way to do DMA from userspace is to use
>> vfio/iommufd.
> I am not sure how this can work in our use-case. We have a very simple
> set of IP blocks implemented in a FPGA, some of those IP blocks are
> able to perform DMA operations. The register of those IP blocks are
> mapped into a user-space application using the existing, accepted
> upstream, UIO subsystem. Some of those registers allow to program DMA
> transfers. So far, we can do all what we need, except program those DMA
> transfers. Lots of people are having the same issue, and zillions of
> ugly out-of-tree solutions flourish all over, and we're trying to see
> if we can constructively find a solution that would be acceptable
> upstream to resolve this use-case. Our platform is an old PowerPC with
> no IOMMU.
Maybe I should try to better explain the concern here. The question is "Where is the source code of your FPGA driver?".
I mean that you are trying to replace the out-of-tree solution is rather welcomed, but the out-of-tree solutions are out-of-tree because they don't fit with requirements to be added to the core Linux tree.
And one of those requirements is that you need to provide the source code of the userspace user of this interface, in this case here that is your FPGA driver. An MIT/X11 license is usually sufficient, GPL is of course seen as better and it must not be a toy application, but rather the real thing.
And that is what people usually don't want and that's why no in-tree solution exists for this.
Regards,
Christian.
>
> Note that the safety argument doesn't hold: UIO already allows to map
> registers into user-space applications, which can already program DMA
> transfers to arbitrary locations in memory. The safety comes from the
> fact that such UIO devices are associated with permissions that the
> system administrator controls, to decide which applications are safe
> enough to be granted access to those UIO devices. Therefore, providing
> access through UIO of the physical address of well-defined DMA buffers
> properly allocated through a sane API is not reducing the safety of
> anything compared to what UIO already allows today.
>
> Best regards,
>
> Thomas
Am 14.04.25 um 10:17 schrieb Thomas Petazzoni:
> Hello Christian,
>
> On Fri, 11 Apr 2025 14:41:56 +0200
> Christian König <christian.koenig(a)amd.com> wrote:
>
>> But anyway please note that when you want to create new UAPI you need
>> to provide an open source user of it. E.g. link to a repository or
>> something similar in the covert letter should do it.
> Could you clarify what is the "reference" user-space user of UIO that
> exists today?
I honestly don't know. I have never looked into UIO before this patch set arrived.
What I mean is that the general rule to justify adding UAPI to the Linux kernel is that you need to have an open source user of that UAPI.
In other words for UIO this means that you *can't* do things like exposing some kernel functionality like DMA-buf through UIO and then write a binary only driver on top of it.
Regards,
Christian.
>
> Thomas
On Thu, Apr 10, 2025 at 04:53:17PM +0200, Bastien Curutchet wrote:
> Hi all,
>
> Many UIO users performing DMA from their UIO device need to access the
> DMA addresses of the allocated buffers. There are out-of-tree drivers
> that allow to do it but nothing in the mainline.
In which case it does not matter at all for mainline.
FYI the proper and safe way to do DMA from userspace is to use
vfio/iommufd.
Am 11.04.25 um 14:44 schrieb Philipp Stanner:
> On Fri, 2025-04-11 at 13:05 +0200, Christian König wrote:
>> Am 11.04.25 um 11:29 schrieb Philipp Stanner:
>>
>>> [SNIP]
>>>
>>> It could be, however, that at the same moment
>>> nouveau_fence_signal() is
>>> removing that entry, holding the appropriate lock.
>>>
>>> So we have a race. Again.
>>>
>>
>> Ah, yes of course. If signaled is called with or without the lock is
>> actually undetermined.
>>
>>
>>>
>>> You see, fixing things in Nouveau is difficult :)
>>> It gets more difficult if you want to clean it up "properly", so it
>>> conforms to rules such as those from dma_fence.
>>>
>>> I have now provided two fixes that both work, but you are not
>>> satisfied
>>> with from the dma_fence-maintainer's perspective. I understand
>>> that,
>>> but please also understand that it's actually not my primary task
>>> to
>>> work on Nouveau. I just have to fix this bug to move on with my
>>> scheduler work.
>>>
>>
>> Well I'm happy with whatever solution as long as it works, but as
>> far as I can see the approach with the callback simply doesn't.
>>
>> You just can't drop the fence reference for the list from the
>> callback.
>>
>>
>>>
>>> So if you have another idea, feel free to share it. But I'd like to
>>> know how we can go on here.
>>>
>>
>> Well the fence code actually works, doesn't it? The problem is
>> rather that setting the error throws a warning because it doesn't
>> expect signaled fences on the pending list.
>>
>> Maybe we should fix that instead.
> The fence code works as the author intended, but I would be happy if it
> were more explicitly documented.
>
> Regarding the WARN_ON: It occurs in dma_fence_set_error() because there
> is an attempt to set an error code on a signaled fence. I don't think
> that should be "fixed", it works as intended: You must not set an error
> code of a fence that was already signaled.
>
> The reason seems to be that once a fence is signaled, a third party
> might evaluate the error code.
Yeah, more or less correct. The idea is you can't declare an operation as having an error after the operation has already completed.
Because everyone will just wait for the completion and nobody checks the status again after that.
>
> But I think this wasn't wat you meant with "fix".
The idea was to avoid calling dma_fence_set_error() on already signaled fences. Something like this:
@@ -90,7 +90,7 @@ nouveau_fence_context_kill(struct nouveau_fence_chan *fctx, int error)
while (!list_empty(&fctx->pending)) {
fence = list_entry(fctx->pending.next, typeof(*fence), head);
- if (error)
+ if (error & !dma_fence_is_signaled_locked(&fence->base))
dma_fence_set_error(&fence->base, error);
if (nouveau_fence_signal(fence))
That would also improve the handling quite a bit since we now don't set errors on fences which are already completed even if we haven't realized that they are already completed yet.
> In any case, there must not be signaled fences in nouveau's pending-
> list. They must be removed immediately once they signal, and this must
> not race.
Why actually? As far as I can see the pending list is not for the unsignaled fences, but rather the pending interrupt processing.
So having signaled fences on the pending list is perfectly possible.
Regards,
Christian.
>
>>
>>
>>>
>>> I'm running out of ideas. What I'm wondering if we couldn't just
>>> remove
>>> performance hacky fastpath functions such as
>>> nouveau_fence_is_signaled() completely. It seems redundant to me.
>>>
>>
>> That would work for me as well.
> I'll test this approach. Seems a bit like the nuclear approach, but if
> it works we'd at least clean up a lot of this mess.
>
>
> P.
>
>
>>
>>
>>>
>>>
>>> Or we might add locking to it, but IDK what was achieved with RCU
>>> here.
>>> In any case it's definitely bad that Nouveau has so many redundant
>>> and
>>> half-redundant mechanisms.
>>>
>>
>> Yeah, agree messing with the locks even more won't help us here.
>>
>> Regards,
>> Christian.
>>
>>
>>>
>>>
>>>
>>> P.
>>>
>>>
>>>>
>>>>
>>>> P.
>>>>
>>>>
>>>>>
>>>>> Regards,
>>>>> Christian.
>>>>>
>>>>>
>>>>>>
>>>>>> P.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Christian.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Replace the call to dma_fence_is_signaled() with
>>>>>>>> nouveau_fence_base_is_signaled().
>>>>>>>>
>>>>>>>> Cc: <stable(a)vger.kernel.org> # 4.10+, precise commit not
>>>>>>>> to
>>>>>>>> be
>>>>>>>> determined
>>>>>>>> Signed-off-by: Philipp Stanner <phasta(a)kernel.org>
>>>>>>>> ---
>>>>>>>> drivers/gpu/drm/nouveau/nouveau_fence.c | 2 +-
>>>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>>>
>>>>>>>> diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> index 7cc84472cece..33535987d8ed 100644
>>>>>>>> --- a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> @@ -274,7 +274,7 @@ nouveau_fence_done(struct
>>>>>>>> nouveau_fence
>>>>>>>> *fence)
>>>>>>>> nvif_event_block(&fctx->event);
>>>>>>>> spin_unlock_irqrestore(&fctx->lock,
>>>>>>>> flags);
>>>>>>>> }
>>>>>>>> - return dma_fence_is_signaled(&fence->base);
>>>>>>>> + return test_bit(DMA_FENCE_FLAG_SIGNALED_BIT,
>>>>>>>> &fence-
>>>>>>>>
>>>>>>>>>
>>>>>>>>> base.flags);
>>>>>>>>>
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> static long
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>