The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink. In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.
Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.
For instance, in the following sequence: 1) of_platform_depopulate() 2) of_overlay_remove()
During the step 1, devices are released and related devlinks are removed (jobs pushed in the workqueue). During the step 2, OF nodes are destroyed but, without any synchronisation with devlink removal jobs, of_overlay_remove() can raise warnings related to missing of_node_put(): ERROR: memory leak, expected refcount 1 instead of 2
Indeed, the missing of_node_put() call is going to be done, too late, from the workqueue job execution.
Introduce device_link_wait_removal() to offer a way to synchronize operations waiting for the end of devlink removals (i.e. end of workqueue jobs). Also, as a flushing operation is done on the workqueue, the workqueue used is moved from a system-wide workqueue to a local one.
Cc: stable@vger.kernel.org Signed-off-by: Herve Codina herve.codina@bootlin.com Tested-by: Luca Ceresoli luca.ceresoli@bootlin.com Reviewed-by: Nuno Sa nuno.sa@analog.com --- drivers/base/core.c | 26 +++++++++++++++++++++++--- include/linux/device.h | 1 + 2 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/drivers/base/core.c b/drivers/base/core.c index d5f4e4aac09b..48b28c59c592 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void); static void __fw_devlink_link_to_consumers(struct device *dev); static bool fw_devlink_drv_reg_done; static bool fw_devlink_best_effort; +static struct workqueue_struct *device_link_wq;
/** * __fwnode_link_add - Create a link between two fwnode_handles. @@ -532,12 +533,26 @@ static void devlink_dev_release(struct device *dev) /* * It may take a while to complete this work because of the SRCU * synchronization in device_link_release_fn() and if the consumer or - * supplier devices get deleted when it runs, so put it into the "long" - * workqueue. + * supplier devices get deleted when it runs, so put it into the + * dedicated workqueue. */ - queue_work(system_long_wq, &link->rm_work); + queue_work(device_link_wq, &link->rm_work); }
+/** + * device_link_wait_removal - Wait for ongoing devlink removal jobs to terminate + */ +void device_link_wait_removal(void) +{ + /* + * devlink removal jobs are queued in the dedicated work queue. + * To be sure that all removal jobs are terminated, ensure that any + * scheduled work has run to completion. + */ + flush_workqueue(device_link_wq); +} +EXPORT_SYMBOL_GPL(device_link_wait_removal); + static struct class devlink_class = { .name = "devlink", .dev_groups = devlink_groups, @@ -4099,9 +4114,14 @@ int __init devices_init(void) sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj); if (!sysfs_dev_char_kobj) goto char_kobj_err; + device_link_wq = alloc_workqueue("device_link_wq", 0, 0); + if (!device_link_wq) + goto wq_err;
return 0;
+ wq_err: + kobject_put(sysfs_dev_char_kobj); char_kobj_err: kobject_put(sysfs_dev_block_kobj); block_kobj_err: diff --git a/include/linux/device.h b/include/linux/device.h index 1795121dee9a..d7d8305a72e8 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -1249,6 +1249,7 @@ void device_link_del(struct device_link *link); void device_link_remove(void *consumer, struct device *supplier); void device_links_supplier_sync_state_pause(void); void device_links_supplier_sync_state_resume(void); +void device_link_wait_removal(void);
/* Create alias, so I can be autoloaded. */ #define MODULE_ALIAS_CHARDEV(major,minor) \
Hi Herve,
On Thu, 2024-03-07 at 12:10 +0100, Herve Codina wrote:
The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink. In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.
Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.
For instance, in the following sequence: 1) of_platform_depopulate() 2) of_overlay_remove()
During the step 1, devices are released and related devlinks are removed (jobs pushed in the workqueue). During the step 2, OF nodes are destroyed but, without any synchronisation with devlink removal jobs, of_overlay_remove() can raise warnings related to missing of_node_put(): ERROR: memory leak, expected refcount 1 instead of 2
Indeed, the missing of_node_put() call is going to be done, too late, from the workqueue job execution.
Introduce device_link_wait_removal() to offer a way to synchronize operations waiting for the end of devlink removals (i.e. end of workqueue jobs). Also, as a flushing operation is done on the workqueue, the workqueue used is moved from a system-wide workqueue to a local one.
Cc: stable@vger.kernel.org Signed-off-by: Herve Codina herve.codina@bootlin.com Tested-by: Luca Ceresoli luca.ceresoli@bootlin.com Reviewed-by: Nuno Sa nuno.sa@analog.com
drivers/base/core.c | 26 +++++++++++++++++++++++--- include/linux/device.h | 1 + 2 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/drivers/base/core.c b/drivers/base/core.c index d5f4e4aac09b..48b28c59c592 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void); static void __fw_devlink_link_to_consumers(struct device *dev); static bool fw_devlink_drv_reg_done; static bool fw_devlink_best_effort; +static struct workqueue_struct *device_link_wq; /** * __fwnode_link_add - Create a link between two fwnode_handles. @@ -532,12 +533,26 @@ static void devlink_dev_release(struct device *dev) /* * It may take a while to complete this work because of the SRCU * synchronization in device_link_release_fn() and if the consumer or
* supplier devices get deleted when it runs, so put it into the "long"* workqueue.
* supplier devices get deleted when it runs, so put it into the* dedicated workqueue.*/
- queue_work(system_long_wq, &link->rm_work);
- queue_work(device_link_wq, &link->rm_work);
} +/**
- device_link_wait_removal - Wait for ongoing devlink removal jobs to terminate
- */
+void device_link_wait_removal(void) +{
- /*
* devlink removal jobs are queued in the dedicated work queue.* To be sure that all removal jobs are terminated, ensure that any* scheduled work has run to completion.*/- flush_workqueue(device_link_wq);
+} +EXPORT_SYMBOL_GPL(device_link_wait_removal);
static struct class devlink_class = { .name = "devlink", .dev_groups = devlink_groups, @@ -4099,9 +4114,14 @@ int __init devices_init(void) sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj); if (!sysfs_dev_char_kobj) goto char_kobj_err;
- device_link_wq = alloc_workqueue("device_link_wq", 0, 0);
My rb tag was with the assumption this is moved into devlink_class_init(). IIUC, Saravana also agreed with that [1]. But it looks like he missed that we are allocating the queue in devices_init() and not in devlink_class_init().
I'm also not sure if this is in line with what Rafael wanted for ccing stable. How do we know the next patch depends on this one?
[1]: https://lore.kernel.org/lkml/CAGETcx_gNWOTsSZMaZu+XU1-5Z60WEcMhw08t4Sn_-YgkC...
- Nuno Sá
On Thu, 2024-03-07 at 12:50 +0100, Nuno Sá wrote:
Hi Herve,
On Thu, 2024-03-07 at 12:10 +0100, Herve Codina wrote:
The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink. In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.
Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.
For instance, in the following sequence: 1) of_platform_depopulate() 2) of_overlay_remove()
During the step 1, devices are released and related devlinks are removed (jobs pushed in the workqueue). During the step 2, OF nodes are destroyed but, without any synchronisation with devlink removal jobs, of_overlay_remove() can raise warnings related to missing of_node_put(): ERROR: memory leak, expected refcount 1 instead of 2
Indeed, the missing of_node_put() call is going to be done, too late, from the workqueue job execution.
Introduce device_link_wait_removal() to offer a way to synchronize operations waiting for the end of devlink removals (i.e. end of workqueue jobs). Also, as a flushing operation is done on the workqueue, the workqueue used is moved from a system-wide workqueue to a local one.
Cc: stable@vger.kernel.org Signed-off-by: Herve Codina herve.codina@bootlin.com Tested-by: Luca Ceresoli luca.ceresoli@bootlin.com Reviewed-by: Nuno Sa nuno.sa@analog.com
drivers/base/core.c | 26 +++++++++++++++++++++++--- include/linux/device.h | 1 + 2 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/drivers/base/core.c b/drivers/base/core.c index d5f4e4aac09b..48b28c59c592 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void); static void __fw_devlink_link_to_consumers(struct device *dev); static bool fw_devlink_drv_reg_done; static bool fw_devlink_best_effort; +static struct workqueue_struct *device_link_wq; /** * __fwnode_link_add - Create a link between two fwnode_handles. @@ -532,12 +533,26 @@ static void devlink_dev_release(struct device *dev) /* * It may take a while to complete this work because of the SRCU * synchronization in device_link_release_fn() and if the consumer or
* supplier devices get deleted when it runs, so put it into the "long"* workqueue.
* supplier devices get deleted when it runs, so put it into the* dedicated workqueue.*/
- queue_work(system_long_wq, &link->rm_work);
- queue_work(device_link_wq, &link->rm_work);
} +/**
- device_link_wait_removal - Wait for ongoing devlink removal jobs to terminate
- */
+void device_link_wait_removal(void) +{
- /*
* devlink removal jobs are queued in the dedicated work queue.* To be sure that all removal jobs are terminated, ensure that any* scheduled work has run to completion.*/- flush_workqueue(device_link_wq);
+} +EXPORT_SYMBOL_GPL(device_link_wait_removal);
static struct class devlink_class = { .name = "devlink", .dev_groups = devlink_groups, @@ -4099,9 +4114,14 @@ int __init devices_init(void) sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj); if (!sysfs_dev_char_kobj) goto char_kobj_err;
- device_link_wq = alloc_workqueue("device_link_wq", 0, 0);
My rb tag was with the assumption this is moved into devlink_class_init(). IIUC, Saravana also agreed with that [1]. But it looks like he missed that we are allocating the queue in devices_init() and not in devlink_class_init().
I'm also not sure if this is in line with what Rafael wanted for ccing stable. How do we know the next patch depends on this one?
Should have looked before:
https://www.kernel.org/doc/Documentation/process/stable-kernel-rules.rst
So, I think what you have is correct :)
- Nuno Sá
Hi Nuno,
On Thu, 07 Mar 2024 12:50:52 +0100 Nuno Sá noname.nuno@gmail.com wrote:
Hi Herve,
On Thu, 2024-03-07 at 12:10 +0100, Herve Codina wrote:
The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink. In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.
Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.
For instance, in the following sequence: 1) of_platform_depopulate() 2) of_overlay_remove()
During the step 1, devices are released and related devlinks are removed (jobs pushed in the workqueue). During the step 2, OF nodes are destroyed but, without any synchronisation with devlink removal jobs, of_overlay_remove() can raise warnings related to missing of_node_put(): ERROR: memory leak, expected refcount 1 instead of 2
Indeed, the missing of_node_put() call is going to be done, too late, from the workqueue job execution.
Introduce device_link_wait_removal() to offer a way to synchronize operations waiting for the end of devlink removals (i.e. end of workqueue jobs). Also, as a flushing operation is done on the workqueue, the workqueue used is moved from a system-wide workqueue to a local one.
Cc: stable@vger.kernel.org Signed-off-by: Herve Codina herve.codina@bootlin.com Tested-by: Luca Ceresoli luca.ceresoli@bootlin.com Reviewed-by: Nuno Sa nuno.sa@analog.com
drivers/base/core.c | 26 +++++++++++++++++++++++--- include/linux/device.h | 1 + 2 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/drivers/base/core.c b/drivers/base/core.c index d5f4e4aac09b..48b28c59c592 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void); static void __fw_devlink_link_to_consumers(struct device *dev); static bool fw_devlink_drv_reg_done; static bool fw_devlink_best_effort; +static struct workqueue_struct *device_link_wq; /** * __fwnode_link_add - Create a link between two fwnode_handles. @@ -532,12 +533,26 @@ static void devlink_dev_release(struct device *dev) /* * It may take a while to complete this work because of the SRCU * synchronization in device_link_release_fn() and if the consumer or
* supplier devices get deleted when it runs, so put it into the "long"* workqueue.
* supplier devices get deleted when it runs, so put it into the* dedicated workqueue.*/
- queue_work(system_long_wq, &link->rm_work);
- queue_work(device_link_wq, &link->rm_work);
} +/**
- device_link_wait_removal - Wait for ongoing devlink removal jobs to terminate
- */
+void device_link_wait_removal(void) +{
- /*
* devlink removal jobs are queued in the dedicated work queue.* To be sure that all removal jobs are terminated, ensure that any* scheduled work has run to completion.*/- flush_workqueue(device_link_wq);
+} +EXPORT_SYMBOL_GPL(device_link_wait_removal);
static struct class devlink_class = { .name = "devlink", .dev_groups = devlink_groups, @@ -4099,9 +4114,14 @@ int __init devices_init(void) sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj); if (!sysfs_dev_char_kobj) goto char_kobj_err;
- device_link_wq = alloc_workqueue("device_link_wq", 0, 0);
My rb tag was with the assumption this is moved into devlink_class_init(). IIUC, Saravana also agreed with that [1]. But it looks like he missed that we are allocating the queue in devices_init() and not in devlink_class_init().
I'm also not sure if this is in line with what Rafael wanted for ccing stable. How do we know the next patch depends on this one?
We discussed that point and I understood that you were ok to do that on your side: https://lore.kernel.org/linux-kernel/f42ceee61ddb8b50c347589649d4131476ab5d8...
Sorry if I misunderstood.
I am going to wait for other comments on this current series before re-sending with our 'Reviewed-by' removed if needed. Let me know.
Best regards, Hervé
On Thu, 2024-03-07 at 13:16 +0100, Herve Codina wrote:
Hi Nuno,
On Thu, 07 Mar 2024 12:50:52 +0100 Nuno Sá noname.nuno@gmail.com wrote:
Hi Herve,
On Thu, 2024-03-07 at 12:10 +0100, Herve Codina wrote:
The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink. In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.
Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.
For instance, in the following sequence: 1) of_platform_depopulate() 2) of_overlay_remove()
During the step 1, devices are released and related devlinks are removed (jobs pushed in the workqueue). During the step 2, OF nodes are destroyed but, without any synchronisation with devlink removal jobs, of_overlay_remove() can raise warnings related to missing of_node_put(): ERROR: memory leak, expected refcount 1 instead of 2
Indeed, the missing of_node_put() call is going to be done, too late, from the workqueue job execution.
Introduce device_link_wait_removal() to offer a way to synchronize operations waiting for the end of devlink removals (i.e. end of workqueue jobs). Also, as a flushing operation is done on the workqueue, the workqueue used is moved from a system-wide workqueue to a local one.
Cc: stable@vger.kernel.org Signed-off-by: Herve Codina herve.codina@bootlin.com Tested-by: Luca Ceresoli luca.ceresoli@bootlin.com Reviewed-by: Nuno Sa nuno.sa@analog.com
drivers/base/core.c | 26 +++++++++++++++++++++++--- include/linux/device.h | 1 + 2 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/drivers/base/core.c b/drivers/base/core.c index d5f4e4aac09b..48b28c59c592 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void); static void __fw_devlink_link_to_consumers(struct device *dev); static bool fw_devlink_drv_reg_done; static bool fw_devlink_best_effort; +static struct workqueue_struct *device_link_wq; /** * __fwnode_link_add - Create a link between two fwnode_handles. @@ -532,12 +533,26 @@ static void devlink_dev_release(struct device *dev) /* * It may take a while to complete this work because of the SRCU * synchronization in device_link_release_fn() and if the consumer or
* supplier devices get deleted when it runs, so put it into the"long"
* workqueue.
* supplier devices get deleted when it runs, so put it into the* dedicated workqueue.*/
- queue_work(system_long_wq, &link->rm_work);
- queue_work(device_link_wq, &link->rm_work);
} +/**
- device_link_wait_removal - Wait for ongoing devlink removal jobs to
terminate
- */
+void device_link_wait_removal(void) +{
- /*
* devlink removal jobs are queued in the dedicated work queue.* To be sure that all removal jobs are terminated, ensure that any* scheduled work has run to completion.*/- flush_workqueue(device_link_wq);
+} +EXPORT_SYMBOL_GPL(device_link_wait_removal);
static struct class devlink_class = { .name = "devlink", .dev_groups = devlink_groups, @@ -4099,9 +4114,14 @@ int __init devices_init(void) sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj); if (!sysfs_dev_char_kobj) goto char_kobj_err;
- device_link_wq = alloc_workqueue("device_link_wq", 0, 0);
My rb tag was with the assumption this is moved into devlink_class_init(). IIUC, Saravana also agreed with that [1]. But it looks like he missed that we are allocating the queue in devices_init() and not in devlink_class_init().
I'm also not sure if this is in line with what Rafael wanted for ccing stable. How do we know the next patch depends on this one?
We discussed that point and I understood that you were ok to do that on your side: https://lore.kernel.org/linux-kernel/f42ceee61ddb8b50c347589649d4131476ab5d8...
Sorry if I misunderstood.
Oh, yeah, I can do that. But given Saravana reply I thought the expectation is to have the queue already allocated in devlink_class_init().
I am going to wait for other comments on this current series before re-sending with our 'Reviewed-by' removed if needed. Let me know.
Anyways, if your expectation was for me to do it later, fine. No need to remove the tag. Sorry for the noise.
- Nuno Sá
linux-stable-mirror@lists.linaro.org
-
Herve Codina -
Nuno Sá