In the following sequence: 1) of_platform_depopulate() 2) of_overlay_remove()
During the step 1, devices are destroyed and devlinks are removed. During the step 2, OF nodes are destroyed but __of_changeset_entry_destroy() can raise warnings related to missing of_node_put(): ERROR: memory leak, expected refcount 1 instead of 2 ...
Indeed, during the devlink removals performed at step 1, the removal itself releasing the device (and the attached of_node) is done by a job queued in a workqueue and so, it is done asynchronously with respect to function calls. When the warning is present, of_node_put() will be called but wrongly too late from the workqueue job.
In order to be sure that any ongoing devlink removals are done before the of_node destruction, synchronize the of_overlay_remove() with the devlink removals.
Fixes: 80dd33cf72d1 ("drivers: base: Fix device link removal") Cc: stable@vger.kernel.org Signed-off-by: Herve Codina herve.codina@bootlin.com --- drivers/of/overlay.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/drivers/of/overlay.c b/drivers/of/overlay.c index 2ae7e9d24a64..7a010a62b9d8 100644 --- a/drivers/of/overlay.c +++ b/drivers/of/overlay.c @@ -8,6 +8,7 @@
#define pr_fmt(fmt) "OF: overlay: " fmt
+#include <linux/device.h> #include <linux/kernel.h> #include <linux/module.h> #include <linux/of.h> @@ -853,6 +854,14 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs) { int i;
+ /* + * Wait for any ongoing device link removals before removing some of + * nodes. Drop the global lock while waiting + */ + mutex_unlock(&of_mutex); + device_link_wait_removal(); + mutex_lock(&of_mutex); + if (ovcs->cset.entries.next) of_changeset_destroy(&ovcs->cset);
@@ -862,7 +871,6 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs) ovcs->id = 0; }
- for (i = 0; i < ovcs->count; i++) { of_node_put(ovcs->fragments[i].target); of_node_put(ovcs->fragments[i].overlay);
On Thu, 2024-02-29 at 11:52 +0100, Herve Codina wrote:
In the following sequence: 1) of_platform_depopulate() 2) of_overlay_remove()
During the step 1, devices are destroyed and devlinks are removed. During the step 2, OF nodes are destroyed but __of_changeset_entry_destroy() can raise warnings related to missing of_node_put(): ERROR: memory leak, expected refcount 1 instead of 2 ...
Indeed, during the devlink removals performed at step 1, the removal itself releasing the device (and the attached of_node) is done by a job queued in a workqueue and so, it is done asynchronously with respect to function calls. When the warning is present, of_node_put() will be called but wrongly too late from the workqueue job.
In order to be sure that any ongoing devlink removals are done before the of_node destruction, synchronize the of_overlay_remove() with the devlink removals.
Fixes: 80dd33cf72d1 ("drivers: base: Fix device link removal") Cc: stable@vger.kernel.org Signed-off-by: Herve Codina herve.codina@bootlin.com
drivers/of/overlay.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/drivers/of/overlay.c b/drivers/of/overlay.c index 2ae7e9d24a64..7a010a62b9d8 100644 --- a/drivers/of/overlay.c +++ b/drivers/of/overlay.c @@ -8,6 +8,7 @@ #define pr_fmt(fmt) "OF: overlay: " fmt +#include <linux/device.h>
This is clearly up to the DT maintainers to decide but, IMHO, I would very much prefer to see fwnode.h included in here rather than directly device.h (so yeah, renaming the function to fwnode_*).
But yeah, I might be biased by own series :)
#include <linux/kernel.h> #include <linux/module.h> #include <linux/of.h> @@ -853,6 +854,14 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs) { int i;
- /*
* Wait for any ongoing device link removals before removing some of* nodes. Drop the global lock while waiting*/- mutex_unlock(&of_mutex);
- device_link_wait_removal();
- mutex_lock(&of_mutex);
I'm still not convinced we need to drop the lock. What happens if someone else grabs the lock while we are in device_link_wait_removal()? Can we guarantee that we can't screw things badly?
The question is, do you have a system/use case where you can really see the deadlock happening? Until I see one, I'm very skeptical about this. And if we have one, I'm not really sure this is also the right solution for it.
- Nuno Sá
On Thu, Feb 29, 2024 at 12:18:49PM +0100, Nuno Sá wrote:
On Thu, 2024-02-29 at 11:52 +0100, Herve Codina wrote:
In the following sequence: 1) of_platform_depopulate() 2) of_overlay_remove()
During the step 1, devices are destroyed and devlinks are removed. During the step 2, OF nodes are destroyed but __of_changeset_entry_destroy() can raise warnings related to missing of_node_put(): ERROR: memory leak, expected refcount 1 instead of 2 ...
Indeed, during the devlink removals performed at step 1, the removal itself releasing the device (and the attached of_node) is done by a job queued in a workqueue and so, it is done asynchronously with respect to function calls. When the warning is present, of_node_put() will be called but wrongly too late from the workqueue job.
In order to be sure that any ongoing devlink removals are done before the of_node destruction, synchronize the of_overlay_remove() with the devlink removals.
Fixes: 80dd33cf72d1 ("drivers: base: Fix device link removal") Cc: stable@vger.kernel.org Signed-off-by: Herve Codina herve.codina@bootlin.com
drivers/of/overlay.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/drivers/of/overlay.c b/drivers/of/overlay.c index 2ae7e9d24a64..7a010a62b9d8 100644 --- a/drivers/of/overlay.c +++ b/drivers/of/overlay.c @@ -8,6 +8,7 @@ #define pr_fmt(fmt) "OF: overlay: " fmt +#include <linux/device.h>
This is clearly up to the DT maintainers to decide but, IMHO, I would very much prefer to see fwnode.h included in here rather than directly device.h (so yeah, renaming the function to fwnode_*).
IMO, the DT code should know almost nothing about fwnode because that's the layer above it. But then overlay stuff is kind of a layer above the core DT code too.
But yeah, I might be biased by own series :)
#include <linux/kernel.h> #include <linux/module.h> #include <linux/of.h> @@ -853,6 +854,14 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs) { int i;
- /*
* Wait for any ongoing device link removals before removing some of* nodes. Drop the global lock while waiting*/- mutex_unlock(&of_mutex);
- device_link_wait_removal();
- mutex_lock(&of_mutex);
I'm still not convinced we need to drop the lock. What happens if someone else grabs the lock while we are in device_link_wait_removal()? Can we guarantee that we can't screw things badly?
It is also just ugly because it's the callers of free_overlay_changeset() that hold the lock and now we're releasing it behind their back.
As device_link_wait_removal() is called before we touch anything, can't it be called before we take the lock? And do we need to call it if applying the overlay fails?
Rob
On Mon, 2024-03-04 at 09:22 -0600, Rob Herring wrote:
On Thu, Feb 29, 2024 at 12:18:49PM +0100, Nuno Sá wrote:
On Thu, 2024-02-29 at 11:52 +0100, Herve Codina wrote:
In the following sequence: 1) of_platform_depopulate() 2) of_overlay_remove()
During the step 1, devices are destroyed and devlinks are removed. During the step 2, OF nodes are destroyed but __of_changeset_entry_destroy() can raise warnings related to missing of_node_put(): ERROR: memory leak, expected refcount 1 instead of 2 ...
Indeed, during the devlink removals performed at step 1, the removal itself releasing the device (and the attached of_node) is done by a job queued in a workqueue and so, it is done asynchronously with respect to function calls. When the warning is present, of_node_put() will be called but wrongly too late from the workqueue job.
In order to be sure that any ongoing devlink removals are done before the of_node destruction, synchronize the of_overlay_remove() with the devlink removals.
Fixes: 80dd33cf72d1 ("drivers: base: Fix device link removal") Cc: stable@vger.kernel.org Signed-off-by: Herve Codina herve.codina@bootlin.com
drivers/of/overlay.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/drivers/of/overlay.c b/drivers/of/overlay.c index 2ae7e9d24a64..7a010a62b9d8 100644 --- a/drivers/of/overlay.c +++ b/drivers/of/overlay.c @@ -8,6 +8,7 @@ #define pr_fmt(fmt) "OF: overlay: " fmt +#include <linux/device.h>
This is clearly up to the DT maintainers to decide but, IMHO, I would very much prefer to see fwnode.h included in here rather than directly device.h (so yeah, renaming the function to fwnode_*).
IMO, the DT code should know almost nothing about fwnode because that's the layer above it. But then overlay stuff is kind of a layer above the core DT code too.
Yeah, my reasoning is just that it may be better than knowing about device.h code... But maybe I'm wrong :)
But yeah, I might be biased by own series :)
#include <linux/kernel.h> #include <linux/module.h> #include <linux/of.h> @@ -853,6 +854,14 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs) { int i;
- /*
* Wait for any ongoing device link removals before removing someof
* nodes. Drop the global lock while waiting*/- mutex_unlock(&of_mutex);
- device_link_wait_removal();
- mutex_lock(&of_mutex);
I'm still not convinced we need to drop the lock. What happens if someone else grabs the lock while we are in device_link_wait_removal()? Can we guarantee that we can't screw things badly?
It is also just ugly because it's the callers of free_overlay_changeset() that hold the lock and now we're releasing it behind their back.
As device_link_wait_removal() is called before we touch anything, can't it be called before we take the lock? And do we need to call it if applying the overlay fails?
My natural feeling was to put it right before checking the node refcount... and I would like to still see proof that there's any potential deadlock. I did not checked the code but the issue with calling it before we take the lock is that likely the device links wont be removed because the overlay removal path (which unbinds devices from drivers) needs to run under the lock?
- Nuno Sá
Hi Rob,
On Mon, 4 Mar 2024 09:22:02 -0600 Rob Herring robh@kernel.org wrote:
...
@@ -853,6 +854,14 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs) { int i;
- /*
* Wait for any ongoing device link removals before removing some of* nodes. Drop the global lock while waiting*/- mutex_unlock(&of_mutex);
- device_link_wait_removal();
- mutex_lock(&of_mutex);
I'm still not convinced we need to drop the lock. What happens if someone else grabs the lock while we are in device_link_wait_removal()? Can we guarantee that we can't screw things badly?
It is also just ugly because it's the callers of free_overlay_changeset() that hold the lock and now we're releasing it behind their back.
As device_link_wait_removal() is called before we touch anything, can't it be called before we take the lock? And do we need to call it if applying the overlay fails?
Indeed, having device_link_wait_removal() is not needed when applying the overlay fails.
I can call device_link_wait_removal() from the caller of_overlay_remove() but not before the lock is taken. We need to call it between __of_changeset_revert_notify() and free_overlay_changeset() and so, the lock is taken.
This lead to the following sequence: --- 8< --- int of_overlay_remove(int *ovcs_id) { ... mutex_lock(&of_mutex); ...
ret = __of_changeset_revert_notify(&ovcs->cset); ...
ret_tmp = overlay_notify(ovcs, OF_OVERLAY_POST_REMOVE); ...
mutex_unlock(&of_mutex); device_link_wait_removal(); mutex_lock(&of_mutex);
free_overlay_changeset(ovcs); ... mutex_unlock(&of_mutex); ... } --- 8< ---
In this sequence, the question is: Do we need to release the mutex lock while device_link_wait_removal() is called ?
Best regards, Hervé
On Mon, Mar 4, 2024 at 8:49 AM Herve Codina herve.codina@bootlin.com wrote:
Hi Rob,
On Mon, 4 Mar 2024 09:22:02 -0600 Rob Herring robh@kernel.org wrote:
...
@@ -853,6 +854,14 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs) { int i;
- /*
- Wait for any ongoing device link removals before removing some of
- nodes. Drop the global lock while waiting
- */
- mutex_unlock(&of_mutex);
- device_link_wait_removal();
- mutex_lock(&of_mutex);
I'm still not convinced we need to drop the lock. What happens if someone else grabs the lock while we are in device_link_wait_removal()? Can we guarantee that we can't screw things badly?
It is also just ugly because it's the callers of free_overlay_changeset() that hold the lock and now we're releasing it behind their back.
As device_link_wait_removal() is called before we touch anything, can't it be called before we take the lock? And do we need to call it if applying the overlay fails?
Rob,
This[1] scenario Luca reported seems like a reason for the device_link_wait_removal() to be where Herve put it. That example seems reasonable.
[1] - https://lore.kernel.org/all/20231220181627.341e8789@booty/
Indeed, having device_link_wait_removal() is not needed when applying the overlay fails.
I can call device_link_wait_removal() from the caller of_overlay_remove() but not before the lock is taken. We need to call it between __of_changeset_revert_notify() and free_overlay_changeset() and so, the lock is taken.
This lead to the following sequence: --- 8< --- int of_overlay_remove(int *ovcs_id) { ... mutex_lock(&of_mutex); ...
ret = __of_changeset_revert_notify(&ovcs->cset); ... ret_tmp = overlay_notify(ovcs, OF_OVERLAY_POST_REMOVE); ... mutex_unlock(&of_mutex); device_link_wait_removal(); mutex_lock(&of_mutex); free_overlay_changeset(ovcs); ... mutex_unlock(&of_mutex); ...} --- 8< ---
In this sequence, the question is: Do we need to release the mutex lock while device_link_wait_removal() is called ?
In general I hate these kinds of sequences that release a lock and then grab it again quickly. It's not always a bug, but my personal take on that is 90% of these introduce a bug.
Drop the unlock/lock and we'll deal a deadlock if we actually hit one. I'm also fairly certain that device_link_wait_removal() can't trigger something else that can cause an OF overlay change while we are in the middle of one. And like Rob said, I'm not sure this unlock/lock is a good solution for that anyway.
Please CC me on the next series. And I'm glad folks convinced you to use flush_workqueue(). As I said in the older series, I think drain_workqueue() will actually break device links.
-Saravana
-Saravana
On Mon, 2024-03-04 at 22:47 -0800, Saravana Kannan wrote:
On Mon, Mar 4, 2024 at 8:49 AM Herve Codina herve.codina@bootlin.com wrote:
Hi Rob,
On Mon, 4 Mar 2024 09:22:02 -0600 Rob Herring robh@kernel.org wrote:
...
@@ -853,6 +854,14 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs) { int i;
- /*
+ * Wait for any ongoing device link removals before removing some of + * nodes. Drop the global lock while waiting + */
- mutex_unlock(&of_mutex);
- device_link_wait_removal();
- mutex_lock(&of_mutex);
I'm still not convinced we need to drop the lock. What happens if someone else grabs the lock while we are in device_link_wait_removal()? Can we guarantee that we can't screw things badly?
It is also just ugly because it's the callers of free_overlay_changeset() that hold the lock and now we're releasing it behind their back.
As device_link_wait_removal() is called before we touch anything, can't it be called before we take the lock? And do we need to call it if applying the overlay fails?
Rob,
This[1] scenario Luca reported seems like a reason for the device_link_wait_removal() to be where Herve put it. That example seems reasonable.
[1] - https://lore.kernel.org/all/20231220181627.341e8789@booty/
I'm still not totally convinced about that. Why not putting the check right before checking the kref in __of_changeset_entry_destroy(). I'll contradict myself a bit because this is just theory but if we look at pci_stop_dev(), which AFAIU, could be reached from a sysfs write(), we have:
device_release_driver(&dev->dev); ... of_pci_remove_node(dev); of_changeset_revert(np->data); of_changeset_destroy(np->data);
So looking at the above we would hit the same issue if we flush the queue in free_overlay_changeset() - as the queue won't be flushed at all and we could have devlink removal due to device_release_driver(). Right?
Again, completely theoretical but seems like a reasonable one plus I'm not understanding the push against having the flush in __of_changeset_entry_destroy(). Conceptually, it looks the best place to me but I may be missing some issue in doing it there?
Indeed, having device_link_wait_removal() is not needed when applying the overlay fails.
I can call device_link_wait_removal() from the caller of_overlay_remove() but not before the lock is taken. We need to call it between __of_changeset_revert_notify() and free_overlay_changeset() and so, the lock is taken.
This lead to the following sequence: --- 8< --- int of_overlay_remove(int *ovcs_id) { ... mutex_lock(&of_mutex); ...
ret = __of_changeset_revert_notify(&ovcs->cset); ...
ret_tmp = overlay_notify(ovcs, OF_OVERLAY_POST_REMOVE); ...
mutex_unlock(&of_mutex); device_link_wait_removal(); mutex_lock(&of_mutex);
free_overlay_changeset(ovcs); ... mutex_unlock(&of_mutex); ... } --- 8< ---
In this sequence, the question is: Do we need to release the mutex lock while device_link_wait_removal() is called ?
In general I hate these kinds of sequences that release a lock and then grab it again quickly. It's not always a bug, but my personal take on that is 90% of these introduce a bug.
Drop the unlock/lock and we'll deal a deadlock if we actually hit one. I'm also fairly certain that device_link_wait_removal() can't trigger something else that can cause an OF overlay change while we are in the middle of one. And like Rob said, I'm not sure this unlock/lock is a good solution for that anyway.
Totally agree. Unless we really see a deadlock this is a very bad idea (IMHO). Even on the PCI code, it seems to me that we're never destroying a changeset from a device/kobj_type release callback. That would be super weird right?
- Nuno Sá
Hi Nuno, Saravana, Rob,
On Tue, 05 Mar 2024 08:36:45 +0100 Nuno Sá noname.nuno@gmail.com wrote:
On Mon, 2024-03-04 at 22:47 -0800, Saravana Kannan wrote:
On Mon, Mar 4, 2024 at 8:49 AM Herve Codina herve.codina@bootlin.com wrote:
Hi Rob,
On Mon, 4 Mar 2024 09:22:02 -0600 Rob Herring robh@kernel.org wrote:
...
@@ -853,6 +854,14 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs) { int i;
- /*
+ * Wait for any ongoing device link removals before removing some of + * nodes. Drop the global lock while waiting + */
- mutex_unlock(&of_mutex);
- device_link_wait_removal();
- mutex_lock(&of_mutex);
I'm still not convinced we need to drop the lock. What happens if someone else grabs the lock while we are in device_link_wait_removal()? Can we guarantee that we can't screw things badly?
It is also just ugly because it's the callers of free_overlay_changeset() that hold the lock and now we're releasing it behind their back.
As device_link_wait_removal() is called before we touch anything, can't it be called before we take the lock? And do we need to call it if applying the overlay fails?
Rob,
This[1] scenario Luca reported seems like a reason for the device_link_wait_removal() to be where Herve put it. That example seems reasonable.
[1] - https://lore.kernel.org/all/20231220181627.341e8789@booty/
I'm still not totally convinced about that. Why not putting the check right before checking the kref in __of_changeset_entry_destroy(). I'll contradict myself a bit because this is just theory but if we look at pci_stop_dev(), which AFAIU, could be reached from a sysfs write(), we have:
device_release_driver(&dev->dev); ... of_pci_remove_node(dev); of_changeset_revert(np->data); of_changeset_destroy(np->data);
So looking at the above we would hit the same issue if we flush the queue in free_overlay_changeset() - as the queue won't be flushed at all and we could have devlink removal due to device_release_driver(). Right?
Again, completely theoretical but seems like a reasonable one plus I'm not understanding the push against having the flush in __of_changeset_entry_destroy(). Conceptually, it looks the best place to me but I may be missing some issue in doing it there?
Instead of having the wait called in __of_changeset_entry_destroy() and so called in a loop. I could move this call in the __of_changeset_entry_destroy() caller (without any of_mutex lock drop).
So this will look like this: --- 8< --- void of_changeset_destroy(struct of_changeset *ocs) { struct of_changeset_entry *ce, *cen;
device_link_wait_removal();
list_for_each_entry_safe_reverse(ce, cen, &ocs->entries, node) __of_changeset_entry_destroy(ce); } --- 8< ---
I already tested on my system and it works correctly with device_link_wait_removal() only called from of_changeset_destroy() as proposed.
Saravana, Nuno, Rob does it seems ok for you ?
...
In general I hate these kinds of sequences that release a lock and then grab it again quickly. It's not always a bug, but my personal take on that is 90% of these introduce a bug.
Drop the unlock/lock and we'll deal a deadlock if we actually hit one. I'm also fairly certain that device_link_wait_removal() can't trigger something else that can cause an OF overlay change while we are in the middle of one. And like Rob said, I'm not sure this unlock/lock is a good solution for that anyway.
Totally agree. Unless we really see a deadlock this is a very bad idea (IMHO). Even on the PCI code, it seems to me that we're never destroying a changeset from a device/kobj_type release callback. That would be super weird right?
Convinced too. I will drop the unlock/re-lock sequence in the next iteration of this series.
Best regards, Hervé
On Tue, 2024-03-05 at 11:27 +0100, Herve Codina wrote:
Hi Nuno, Saravana, Rob,
On Tue, 05 Mar 2024 08:36:45 +0100 Nuno Sá noname.nuno@gmail.com wrote:
On Mon, 2024-03-04 at 22:47 -0800, Saravana Kannan wrote:
On Mon, Mar 4, 2024 at 8:49 AM Herve Codina herve.codina@bootlin.com wrote:
Hi Rob,
On Mon, 4 Mar 2024 09:22:02 -0600 Rob Herring robh@kernel.org wrote:
...
> @@ -853,6 +854,14 @@ static void free_overlay_changeset(struct > overlay_changeset *ovcs) > { > int i; > > + /* > + * Wait for any ongoing device link removals before removing > some of > + * nodes. Drop the global lock while waiting > + */ > + mutex_unlock(&of_mutex); > + device_link_wait_removal(); > + mutex_lock(&of_mutex);
I'm still not convinced we need to drop the lock. What happens if someone else grabs the lock while we are in device_link_wait_removal()? Can we guarantee that we can't screw things badly?
It is also just ugly because it's the callers of free_overlay_changeset() that hold the lock and now we're releasing it behind their back.
As device_link_wait_removal() is called before we touch anything, can't it be called before we take the lock? And do we need to call it if applying the overlay fails?
Rob,
This[1] scenario Luca reported seems like a reason for the device_link_wait_removal() to be where Herve put it. That example seems reasonable.
[1] - https://lore.kernel.org/all/20231220181627.341e8789@booty/
I'm still not totally convinced about that. Why not putting the check right before checking the kref in __of_changeset_entry_destroy(). I'll contradict myself a bit because this is just theory but if we look at pci_stop_dev(), which AFAIU, could be reached from a sysfs write(), we have:
device_release_driver(&dev->dev); ... of_pci_remove_node(dev); of_changeset_revert(np->data); of_changeset_destroy(np->data);
So looking at the above we would hit the same issue if we flush the queue in free_overlay_changeset() - as the queue won't be flushed at all and we could have devlink removal due to device_release_driver(). Right?
Again, completely theoretical but seems like a reasonable one plus I'm not understanding the push against having the flush in __of_changeset_entry_destroy(). Conceptually, it looks the best place to me but I may be missing some issue in doing it there?
Instead of having the wait called in __of_changeset_entry_destroy() and so called in a loop. I could move this call in the __of_changeset_entry_destroy() caller (without any of_mutex lock drop).
Oh, good catch! At this point all the devlinks removals (related to the changeset) should have been queued so yes, we should only need to flush once.
So this will look like this: --- 8< --- void of_changeset_destroy(struct of_changeset *ocs) { struct of_changeset_entry *ce, *cen;
device_link_wait_removal();
list_for_each_entry_safe_reverse(ce, cen, &ocs->entries, node) __of_changeset_entry_destroy(ce); } --- 8< ---
I already tested on my system and it works correctly with device_link_wait_removal() only called from of_changeset_destroy() as proposed.
Saravana, Nuno, Rob does it seems ok for you ?
It looks good to me...
- Nuno Sá
On Tue, Mar 5, 2024 at 2:43 AM Nuno Sá noname.nuno@gmail.com wrote:
On Tue, 2024-03-05 at 11:27 +0100, Herve Codina wrote:
Hi Nuno, Saravana, Rob,
On Tue, 05 Mar 2024 08:36:45 +0100 Nuno Sá noname.nuno@gmail.com wrote:
On Mon, 2024-03-04 at 22:47 -0800, Saravana Kannan wrote:
On Mon, Mar 4, 2024 at 8:49 AM Herve Codina herve.codina@bootlin.com wrote:
Hi Rob,
On Mon, 4 Mar 2024 09:22:02 -0600 Rob Herring robh@kernel.org wrote:
...
> > @@ -853,6 +854,14 @@ static void free_overlay_changeset(struct > > overlay_changeset *ovcs) > > { > > int i; > > > > + /* > > + * Wait for any ongoing device link removals before removing > > some of > > + * nodes. Drop the global lock while waiting > > + */ > > + mutex_unlock(&of_mutex); > > + device_link_wait_removal(); > > + mutex_lock(&of_mutex); > > I'm still not convinced we need to drop the lock. What happens if > someone else > grabs the lock while we are in device_link_wait_removal()? Can we > guarantee that > we can't screw things badly?
It is also just ugly because it's the callers of free_overlay_changeset() that hold the lock and now we're releasing it behind their back.
As device_link_wait_removal() is called before we touch anything, can't it be called before we take the lock? And do we need to call it if applying the overlay fails?
Rob,
This[1] scenario Luca reported seems like a reason for the device_link_wait_removal() to be where Herve put it. That example seems reasonable.
[1] - https://lore.kernel.org/all/20231220181627.341e8789@booty/
I'm still not totally convinced about that. Why not putting the check right before checking the kref in __of_changeset_entry_destroy(). I'll contradict myself a bit because this is just theory but if we look at pci_stop_dev(), which AFAIU, could be reached from a sysfs write(), we have:
device_release_driver(&dev->dev); ... of_pci_remove_node(dev); of_changeset_revert(np->data); of_changeset_destroy(np->data);
So looking at the above we would hit the same issue if we flush the queue in free_overlay_changeset() - as the queue won't be flushed at all and we could have devlink removal due to device_release_driver(). Right?
Again, completely theoretical but seems like a reasonable one plus I'm not understanding the push against having the flush in __of_changeset_entry_destroy(). Conceptually, it looks the best place to me but I may be missing some issue in doing it there?
Instead of having the wait called in __of_changeset_entry_destroy() and so called in a loop. I could move this call in the __of_changeset_entry_destroy() caller (without any of_mutex lock drop).
Oh, good catch! At this point all the devlinks removals (related to the changeset) should have been queued so yes, we should only need to flush once.
So this will look like this: --- 8< --- void of_changeset_destroy(struct of_changeset *ocs) { struct of_changeset_entry *ce, *cen;
device_link_wait_removal(); list_for_each_entry_safe_reverse(ce, cen, &ocs->entries, node) __of_changeset_entry_destroy(ce);} --- 8< ---
I already tested on my system and it works correctly with device_link_wait_removal() only called from of_changeset_destroy() as proposed.
Saravana, Nuno, Rob does it seems ok for you ?
Looks good to me.
-Saravana
It looks good to me...
- Nuno Sá
linux-stable-mirror@lists.linaro.org
-
Herve Codina -
Nuno Sá -
Rob Herring -
Saravana Kannan