On Thu, Nov 29, 2018 at 9:51 AM Logan Gunthorpe logang@deltatee.com wrote:
On 2018-11-29 10:30 a.m., Dan Williams wrote:
Oh! Yes, nice find. We need to wait for the percpu-ref to be dead and all outstanding references dropped before we can proceed to arch_remove_memory(), and I think this problem has been there since day one because the final exit was always after devm_memremap_pages() release which means arch_remove_memory() was always racing any final put_page(). I'll take a look, it seems the arch_remove_pages() call needs to be moved out-of-line to its own context and wait for the final exit of the percpu-ref.
Ok, well I thought moving the wait_for_completion() into the kill() call was a pretty good solution to this.
True, it is...
Though, if we move the arch_remove_pages() into a different context, it *may* help with the problem below...
Glad to see my over-engineered proposal in this case might be good for something...
Though, now that I look at it, the current change in question will be wrong if there are two devm_memremap_pages_release()s to call. Both need to drop their references before we can wait_for_completion() ;(. I guess I need multiple percpu_refs or more complex changes to devm_memremap_pages_release().
Can you just have a normal device-level kref for this case? On final device-level kref_put then kill the percpu_ref? I guess the problem is devm semantics where p2pdma only gets one callback on a driver ->remove() event. I'm not sure how to support multiple references of the same pages without creating a non-devm version of devm_memremap_pages(). I'm not opposed to that, but afaiu I don't think p2pdma is compatible with devm as long as it supports N>1:1 mappings of the same range.
Hmm, no I think you misunderstood what I said. I'm saying I need to have exactly one percpu_ref per call to devm_memremap_pages() and this is doable, just slightly annoying. Right now I have one percpu_ref for multiple calls to devm_memremap_pages() which doesn't work with the above fix because there will always be a wait_for_completion() before the last references are dropped in this way:
- First devm_memremap_pages_release() is called which drops it's
reference and waits_for_completion().
- The second devm_memremap_pages_release() needs to be called to drop
it's reference, but can't seeing the first is waiting, and therefore the percpu_ref never goes to zero and the wait_for_completion() never returns.
Got it, let me see how bad moving arch_remove_memory() turns out, sounds like a decent approach to coordinate multiple users of a single ref.