vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org --- drivers/vhost/vhost.c | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe28..bbaff6a5e21b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i;
for (i = 0; i < dev->nvqs; ++i) { + mutex_lock(&dev->vqs[i]->mutex); if (dev->vqs[i]->error_ctx) eventfd_ctx_put(dev->vqs[i]->error_ctx); if (dev->vqs[i]->kick) @@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) if (dev->vqs[i]->call_ctx.ctx) eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); vhost_vq_reset(dev, dev->vqs[i]); + mutex_unlock(&dev->vqs[i]->mutex); } vhost_dev_free_iovecs(dev); if (dev->log_ctx)
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
This issue is similar to [1] that should be already fixed upstream by [2].
However I think this patch would have prevented some issues, because vhost_vq_reset() sets vq->private to NULL, preventing the worker from running.
Anyway I think that when we enter in vhost_dev_cleanup() the worker should be already stopped, so it shouldn't be necessary to take the mutex. But in order to prevent future issues maybe it's better to take them, so:
Reviewed-by: Stefano Garzarella sgarzare@redhat.com
[1] https://syzkaller.appspot.com/bug?id=993d8b5e64393ed9e6a70f9ae4de0119c605a82... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
drivers/vhost/vhost.c | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe28..bbaff6a5e21b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i;
for (i = 0; i < dev->nvqs; ++i) {
mutex_lock(&dev->vqs[i]->mutex);@@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) if (dev->vqs[i]->call_ctx.ctx) eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); vhost_vq_reset(dev, dev->vqs[i]);
mutex_unlock(&dev->vqs[i]->mutex);-- 2.35.1.574.g5d30c73bfb-goog
On Wed, 02 Mar 2022, Stefano Garzarella wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
This issue is similar to [1] that should be already fixed upstream by [2].
However I think this patch would have prevented some issues, because vhost_vq_reset() sets vq->private to NULL, preventing the worker from running.
Anyway I think that when we enter in vhost_dev_cleanup() the worker should be already stopped, so it shouldn't be necessary to take the mutex. But in order to prevent future issues maybe it's better to take them, so:
Reviewed-by: Stefano Garzarella sgarzare@redhat.com
Thanks for the analysis and the review Stefano.
On Wed, Mar 02, 2022 at 10:34:46AM +0100, Stefano Garzarella wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
This issue is similar to [1] that should be already fixed upstream by [2].
However I think this patch would have prevented some issues, because vhost_vq_reset() sets vq->private to NULL, preventing the worker from running.
Anyway I think that when we enter in vhost_dev_cleanup() the worker should be already stopped, so it shouldn't be necessary to take the mutex. But in order to prevent future issues maybe it's better to take them, so:
Reviewed-by: Stefano Garzarella sgarzare@redhat.com
[1] https://syzkaller.appspot.com/bug?id=993d8b5e64393ed9e6a70f9ae4de0119c605a82... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
Right. I want to queue this but I would like to get a warning so we can detect issues like [2] before they cause more issues.
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
drivers/vhost/vhost.c | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe28..bbaff6a5e21b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i;
for (i = 0; i < dev->nvqs; ++i) {
mutex_lock(&dev->vqs[i]->mutex);@@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) if (dev->vqs[i]->call_ctx.ctx) eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); vhost_vq_reset(dev, dev->vqs[i]);
mutex_unlock(&dev->vqs[i]->mutex);-- 2.35.1.574.g5d30c73bfb-goog
On Wed, Mar 02, 2022 at 08:35:08AM -0500, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 10:34:46AM +0100, Stefano Garzarella wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
This issue is similar to [1] that should be already fixed upstream by [2].
However I think this patch would have prevented some issues, because vhost_vq_reset() sets vq->private to NULL, preventing the worker from running.
Anyway I think that when we enter in vhost_dev_cleanup() the worker should be already stopped, so it shouldn't be necessary to take the mutex. But in order to prevent future issues maybe it's better to take them, so:
Reviewed-by: Stefano Garzarella sgarzare@redhat.com
[1] https://syzkaller.appspot.com/bug?id=993d8b5e64393ed9e6a70f9ae4de0119c605a82... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
Right. I want to queue this but I would like to get a warning so we can detect issues like [2] before they cause more issues.
I agree, what about moving the warning that we already have higher up, right at the beginning of the function?
I mean something like this:
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe2..1721ff3f18c0 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -692,6 +692,8 @@ void vhost_dev_cleanup(struct vhost_dev *dev) { int i;
+ WARN_ON(!llist_empty(&dev->work_list)); + for (i = 0; i < dev->nvqs; ++i) { if (dev->vqs[i]->error_ctx) eventfd_ctx_put(dev->vqs[i]->error_ctx); @@ -712,7 +714,6 @@ void vhost_dev_cleanup(struct vhost_dev *dev) dev->iotlb = NULL; vhost_clear_msg(dev); wake_up_interruptible_poll(&dev->wait, EPOLLIN | EPOLLRDNORM); - WARN_ON(!llist_empty(&dev->work_list)); if (dev->worker) { kthread_stop(dev->worker); dev->worker = NULL;
And maybe we can also check vq->private and warn in the loop, because the work_list may be empty if the device is doing nothing.
Thanks, Stefano
On Wed, Mar 02, 2022 at 03:11:21PM +0100, Stefano Garzarella wrote:
On Wed, Mar 02, 2022 at 08:35:08AM -0500, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 10:34:46AM +0100, Stefano Garzarella wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
This issue is similar to [1] that should be already fixed upstream by [2].
However I think this patch would have prevented some issues, because vhost_vq_reset() sets vq->private to NULL, preventing the worker from running.
Anyway I think that when we enter in vhost_dev_cleanup() the worker should be already stopped, so it shouldn't be necessary to take the mutex. But in order to prevent future issues maybe it's better to take them, so:
Reviewed-by: Stefano Garzarella sgarzare@redhat.com
[1] https://syzkaller.appspot.com/bug?id=993d8b5e64393ed9e6a70f9ae4de0119c605a82... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
Right. I want to queue this but I would like to get a warning so we can detect issues like [2] before they cause more issues.
I agree, what about moving the warning that we already have higher up, right at the beginning of the function?
I mean something like this:
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe2..1721ff3f18c0 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -692,6 +692,8 @@ void vhost_dev_cleanup(struct vhost_dev *dev) { int i;
WARN_ON(!llist_empty(&dev->work_list));for (i = 0; i < dev->nvqs; ++i) { if (dev->vqs[i]->error_ctx) eventfd_ctx_put(dev->vqs[i]->error_ctx);@@ -712,7 +714,6 @@ void vhost_dev_cleanup(struct vhost_dev *dev) dev->iotlb = NULL; vhost_clear_msg(dev); wake_up_interruptible_poll(&dev->wait, EPOLLIN | EPOLLRDNORM);
WARN_ON(!llist_empty(&dev->work_list)); if (dev->worker) { kthread_stop(dev->worker); dev->worker = NULL;
Hmm I'm not sure why it matters.
And maybe we can also check vq->private and warn in the loop, because the work_list may be empty if the device is doing nothing.
Thanks, Stefano
On Wed, Mar 02, 2022 at 09:50:38AM -0500, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 03:11:21PM +0100, Stefano Garzarella wrote:
On Wed, Mar 02, 2022 at 08:35:08AM -0500, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 10:34:46AM +0100, Stefano Garzarella wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
This issue is similar to [1] that should be already fixed upstream by [2].
However I think this patch would have prevented some issues, because vhost_vq_reset() sets vq->private to NULL, preventing the worker from running.
Anyway I think that when we enter in vhost_dev_cleanup() the worker should be already stopped, so it shouldn't be necessary to take the mutex. But in order to prevent future issues maybe it's better to take them, so:
Reviewed-by: Stefano Garzarella sgarzare@redhat.com
[1] https://syzkaller.appspot.com/bug?id=993d8b5e64393ed9e6a70f9ae4de0119c605a82... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
Right. I want to queue this but I would like to get a warning so we can detect issues like [2] before they cause more issues.
I agree, what about moving the warning that we already have higher up, right at the beginning of the function?
I mean something like this:
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe2..1721ff3f18c0 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -692,6 +692,8 @@ void vhost_dev_cleanup(struct vhost_dev *dev) { int i;
WARN_ON(!llist_empty(&dev->work_list));for (i = 0; i < dev->nvqs; ++i) { if (dev->vqs[i]->error_ctx) eventfd_ctx_put(dev->vqs[i]->error_ctx);@@ -712,7 +714,6 @@ void vhost_dev_cleanup(struct vhost_dev *dev) dev->iotlb = NULL; vhost_clear_msg(dev); wake_up_interruptible_poll(&dev->wait, EPOLLIN | EPOLLRDNORM);
WARN_ON(!llist_empty(&dev->work_list)); if (dev->worker) { kthread_stop(dev->worker); dev->worker = NULL;Hmm I'm not sure why it matters.
Because after this new patch, putting locks in the while loop, when we finish the loop the workers should be stopped, because vhost_vq_reset() sets vq->private to NULL.
But the best thing IMHO is to check that there is no backend set for each vq, so the workers have been stopped correctly at this point.
Thanks, Stefano
On Wed, Mar 02, 2022 at 04:36:43PM +0100, Stefano Garzarella wrote:
On Wed, Mar 02, 2022 at 09:50:38AM -0500, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 03:11:21PM +0100, Stefano Garzarella wrote:
On Wed, Mar 02, 2022 at 08:35:08AM -0500, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 10:34:46AM +0100, Stefano Garzarella wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
This issue is similar to [1] that should be already fixed upstream by [2].
However I think this patch would have prevented some issues, because vhost_vq_reset() sets vq->private to NULL, preventing the worker from running.
Anyway I think that when we enter in vhost_dev_cleanup() the worker should be already stopped, so it shouldn't be necessary to take the mutex. But in order to prevent future issues maybe it's better to take them, so:
Reviewed-by: Stefano Garzarella sgarzare@redhat.com
[1] https://syzkaller.appspot.com/bug?id=993d8b5e64393ed9e6a70f9ae4de0119c605a82... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
Right. I want to queue this but I would like to get a warning so we can detect issues like [2] before they cause more issues.
I agree, what about moving the warning that we already have higher up, right at the beginning of the function?
I mean something like this:
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe2..1721ff3f18c0 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -692,6 +692,8 @@ void vhost_dev_cleanup(struct vhost_dev *dev) { int i;
WARN_ON(!llist_empty(&dev->work_list));for (i = 0; i < dev->nvqs; ++i) { if (dev->vqs[i]->error_ctx) eventfd_ctx_put(dev->vqs[i]->error_ctx);@@ -712,7 +714,6 @@ void vhost_dev_cleanup(struct vhost_dev *dev) dev->iotlb = NULL; vhost_clear_msg(dev); wake_up_interruptible_poll(&dev->wait, EPOLLIN | EPOLLRDNORM);
WARN_ON(!llist_empty(&dev->work_list)); if (dev->worker) { kthread_stop(dev->worker); dev->worker = NULL;Hmm I'm not sure why it matters.
Because after this new patch, putting locks in the while loop, when we finish the loop the workers should be stopped, because vhost_vq_reset() sets vq->private to NULL.
But the best thing IMHO is to check that there is no backend set for each vq, so the workers have been stopped correctly at this point.
Thanks, Stefano
It's the list of workers waiting to run though. That is not affected by vq lock at all.
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
drivers/vhost/vhost.c | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe28..bbaff6a5e21b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i; for (i = 0; i < dev->nvqs; ++i) {
mutex_lock(&dev->vqs[i]->mutex);@@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) if (dev->vqs[i]->call_ctx.ctx) eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); vhost_vq_reset(dev, dev->vqs[i]);
mutex_unlock(&dev->vqs[i]->mutex);
So this is a mitigation plan but the bug is still there though we don't know exactly what it is. I would prefer adding something like WARN_ON(mutex_is_locked(vqs[i]->mutex) here - does this make sense?
vhost_dev_free_iovecs(dev); if (dev->log_ctx) -- 2.35.1.574.g5d30c73bfb-goog
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
drivers/vhost/vhost.c | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe28..bbaff6a5e21b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i; for (i = 0; i < dev->nvqs; ++i) {
mutex_lock(&dev->vqs[i]->mutex);@@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) if (dev->vqs[i]->call_ctx.ctx) eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); vhost_vq_reset(dev, dev->vqs[i]);
mutex_unlock(&dev->vqs[i]->mutex);So this is a mitigation plan but the bug is still there though we don't know exactly what it is. I would prefer adding something like WARN_ON(mutex_is_locked(vqs[i]->mutex) here - does this make sense?
As a rework to this, or as a subsequent patch?
Just before the first lock I assume?
On Wed, Mar 02, 2022 at 01:56:35PM +0000, Lee Jones wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
drivers/vhost/vhost.c | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe28..bbaff6a5e21b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i; for (i = 0; i < dev->nvqs; ++i) {
mutex_lock(&dev->vqs[i]->mutex);@@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) if (dev->vqs[i]->call_ctx.ctx) eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); vhost_vq_reset(dev, dev->vqs[i]);
mutex_unlock(&dev->vqs[i]->mutex);So this is a mitigation plan but the bug is still there though we don't know exactly what it is. I would prefer adding something like WARN_ON(mutex_is_locked(vqs[i]->mutex) here - does this make sense?
As a rework to this, or as a subsequent patch?
Can be a separate patch.
Just before the first lock I assume?
I guess so, yes.
-- Lee Jones [李琼斯] Principal Technical Lead - Developer Services Linaro.org │ Open source software for Arm SoCs Follow Linaro: Facebook | Twitter | Blog
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 01:56:35PM +0000, Lee Jones wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
drivers/vhost/vhost.c | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe28..bbaff6a5e21b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i; for (i = 0; i < dev->nvqs; ++i) {
mutex_lock(&dev->vqs[i]->mutex);@@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) if (dev->vqs[i]->call_ctx.ctx) eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); vhost_vq_reset(dev, dev->vqs[i]);
mutex_unlock(&dev->vqs[i]->mutex);So this is a mitigation plan but the bug is still there though we don't know exactly what it is. I would prefer adding something like WARN_ON(mutex_is_locked(vqs[i]->mutex) here - does this make sense?
As a rework to this, or as a subsequent patch?
Can be a separate patch.
Just before the first lock I assume?
I guess so, yes.
No problem. Patch to follow.
I'm also going to attempt to debug the root cause, but I'm new to this subsystem to it might take a while for me to get my head around.
On Wed, Mar 2, 2022 at 3:57 PM Lee Jones lee.jones@linaro.org wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 01:56:35PM +0000, Lee Jones wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
drivers/vhost/vhost.c | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe28..bbaff6a5e21b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i;
for (i = 0; i < dev->nvqs; ++i) {
mutex_lock(&dev->vqs[i]->mutex); if (dev->vqs[i]->error_ctx) eventfd_ctx_put(dev->vqs[i]->error_ctx); if (dev->vqs[i]->kick)@@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) if (dev->vqs[i]->call_ctx.ctx) eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); vhost_vq_reset(dev, dev->vqs[i]);
mutex_unlock(&dev->vqs[i]->mutex); }So this is a mitigation plan but the bug is still there though we don't know exactly what it is. I would prefer adding something like WARN_ON(mutex_is_locked(vqs[i]->mutex) here - does this make sense?
As a rework to this, or as a subsequent patch?
Can be a separate patch.
Just before the first lock I assume?
I guess so, yes.
No problem. Patch to follow.
I'm also going to attempt to debug the root cause, but I'm new to this subsystem to it might take a while for me to get my head around.
IIUC the root cause should be the same as the one we solved here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
The worker was not stopped before calling vhost_dev_cleanup(). So while the worker was still running we were going to free memory or initialize fields while it was still using virtqueue.
Cheers, Stefano
On Wed, Mar 02, 2022 at 05:28:31PM +0100, Stefano Garzarella wrote:
On Wed, Mar 2, 2022 at 3:57 PM Lee Jones lee.jones@linaro.org wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 01:56:35PM +0000, Lee Jones wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
drivers/vhost/vhost.c | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe28..bbaff6a5e21b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i;
for (i = 0; i < dev->nvqs; ++i) {
mutex_lock(&dev->vqs[i]->mutex); if (dev->vqs[i]->error_ctx) eventfd_ctx_put(dev->vqs[i]->error_ctx); if (dev->vqs[i]->kick)@@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) if (dev->vqs[i]->call_ctx.ctx) eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); vhost_vq_reset(dev, dev->vqs[i]);
mutex_unlock(&dev->vqs[i]->mutex); }So this is a mitigation plan but the bug is still there though we don't know exactly what it is. I would prefer adding something like WARN_ON(mutex_is_locked(vqs[i]->mutex) here - does this make sense?
As a rework to this, or as a subsequent patch?
Can be a separate patch.
Just before the first lock I assume?
I guess so, yes.
No problem. Patch to follow.
I'm also going to attempt to debug the root cause, but I'm new to this subsystem to it might take a while for me to get my head around.
IIUC the root cause should be the same as the one we solved here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
The worker was not stopped before calling vhost_dev_cleanup(). So while the worker was still running we were going to free memory or initialize fields while it was still using virtqueue.
Cheers, Stefano
Right, and I agree but it's not the root though, we do attempt to stop all workers.
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 05:28:31PM +0100, Stefano Garzarella wrote:
On Wed, Mar 2, 2022 at 3:57 PM Lee Jones lee.jones@linaro.org wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 01:56:35PM +0000, Lee Jones wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote: > vhost_vsock_handle_tx_kick() already holds the mutex during its call > to vhost_get_vq_desc(). All we have to do is take the same lock > during virtqueue clean-up and we mitigate the reported issues. > > Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00 > > Cc: stable@vger.kernel.org > Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com > Signed-off-by: Lee Jones lee.jones@linaro.org > --- > drivers/vhost/vhost.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > index 59edb5a1ffe28..bbaff6a5e21b8 100644 > --- a/drivers/vhost/vhost.c > +++ b/drivers/vhost/vhost.c > @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > int i; > > for (i = 0; i < dev->nvqs; ++i) { > + mutex_lock(&dev->vqs[i]->mutex); > if (dev->vqs[i]->error_ctx) > eventfd_ctx_put(dev->vqs[i]->error_ctx); > if (dev->vqs[i]->kick) > @@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > if (dev->vqs[i]->call_ctx.ctx) > eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); > vhost_vq_reset(dev, dev->vqs[i]); > + mutex_unlock(&dev->vqs[i]->mutex); > }
So this is a mitigation plan but the bug is still there though we don't know exactly what it is. I would prefer adding something like WARN_ON(mutex_is_locked(vqs[i]->mutex) here - does this make sense?
As a rework to this, or as a subsequent patch?
Can be a separate patch.
Just before the first lock I assume?
I guess so, yes.
No problem. Patch to follow.
I'm also going to attempt to debug the root cause, but I'm new to this subsystem to it might take a while for me to get my head around.
IIUC the root cause should be the same as the one we solved here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
The worker was not stopped before calling vhost_dev_cleanup(). So while the worker was still running we were going to free memory or initialize fields while it was still using virtqueue.
Right, and I agree but it's not the root though, we do attempt to stop all workers.
Exactly. This is what happens, but the question I'm going to attempt to answer is *why* does this happen.
On Wed, Mar 02, 2022 at 04:49:17PM +0000, Lee Jones wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 05:28:31PM +0100, Stefano Garzarella wrote:
On Wed, Mar 2, 2022 at 3:57 PM Lee Jones lee.jones@linaro.org wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 01:56:35PM +0000, Lee Jones wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
> On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote: > > vhost_vsock_handle_tx_kick() already holds the mutex during its call > > to vhost_get_vq_desc(). All we have to do is take the same lock > > during virtqueue clean-up and we mitigate the reported issues. > > > > Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00 > > > > Cc: stable@vger.kernel.org > > Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com > > Signed-off-by: Lee Jones lee.jones@linaro.org > > --- > > drivers/vhost/vhost.c | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > > index 59edb5a1ffe28..bbaff6a5e21b8 100644 > > --- a/drivers/vhost/vhost.c > > +++ b/drivers/vhost/vhost.c > > @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > int i; > > > > for (i = 0; i < dev->nvqs; ++i) { > > + mutex_lock(&dev->vqs[i]->mutex); > > if (dev->vqs[i]->error_ctx) > > eventfd_ctx_put(dev->vqs[i]->error_ctx); > > if (dev->vqs[i]->kick) > > @@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > if (dev->vqs[i]->call_ctx.ctx) > > eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); > > vhost_vq_reset(dev, dev->vqs[i]); > > + mutex_unlock(&dev->vqs[i]->mutex); > > } > > So this is a mitigation plan but the bug is still there though > we don't know exactly what it is. I would prefer adding something like > WARN_ON(mutex_is_locked(vqs[i]->mutex) here - does this make sense?
As a rework to this, or as a subsequent patch?
Can be a separate patch.
Just before the first lock I assume?
I guess so, yes.
No problem. Patch to follow.
I'm also going to attempt to debug the root cause, but I'm new to this subsystem to it might take a while for me to get my head around.
IIUC the root cause should be the same as the one we solved here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
The worker was not stopped before calling vhost_dev_cleanup(). So while the worker was still running we were going to free memory or initialize fields while it was still using virtqueue.
Right, and I agree but it's not the root though, we do attempt to stop all workers.
Exactly. This is what happens, but the question I'm going to attempt to answer is *why* does this happen.
IIUC the worker was still running because the /dev/vhost-vsock file was not explicitly closed, so vhost_vsock_dev_release() was called in the do_exit() of the process.
In that case there was the issue, because vhost_dev_check_owner() returned false in vhost_vsock_stop() since current->mm was NULL. So it returned earlier, without calling vhost_vq_set_backend(vq, NULL).
This did not stop the worker from continuing to run, causing the multiple issues we are seeing.
current->mm was NULL, because in the do_exit() the address space is cleaned in the exit_mm(), which is called before releasing the files into the exit_task_work().
This can be seen from the logs, where we see first the warnings printed by vhost_dev_cleanup() and then the panic in the worker (e.g. here https://syzkaller.appspot.com/text?tag=CrashLog&x=16a61fce700000)
Mike also added a few more helpful details in this thread: https://lore.kernel.org/virtualization/20220221100500.2x3s2sddqahgdfyt@sgarz...
Thanks, Stefano
On Wed, 02 Mar 2022, Stefano Garzarella wrote:
On Wed, Mar 02, 2022 at 04:49:17PM +0000, Lee Jones wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 05:28:31PM +0100, Stefano Garzarella wrote:
On Wed, Mar 2, 2022 at 3:57 PM Lee Jones lee.jones@linaro.org wrote:
On Wed, 02 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 01:56:35PM +0000, Lee Jones wrote: > On Wed, 02 Mar 2022, Michael S. Tsirkin wrote: > > > On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote: > > > vhost_vsock_handle_tx_kick() already holds the mutex during its call > > > to vhost_get_vq_desc(). All we have to do is take the same lock > > > during virtqueue clean-up and we mitigate the reported issues. > > > > > > Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00 > > > > > > Cc: stable@vger.kernel.org > > > Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com > > > Signed-off-by: Lee Jones lee.jones@linaro.org > > > --- > > > drivers/vhost/vhost.c | 2 ++ > > > 1 file changed, 2 insertions(+) > > > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > > > index 59edb5a1ffe28..bbaff6a5e21b8 100644 > > > --- a/drivers/vhost/vhost.c > > > +++ b/drivers/vhost/vhost.c > > > @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > > int i; > > > > > > for (i = 0; i < dev->nvqs; ++i) { > > > + mutex_lock(&dev->vqs[i]->mutex); > > > if (dev->vqs[i]->error_ctx) > > > eventfd_ctx_put(dev->vqs[i]->error_ctx); > > > if (dev->vqs[i]->kick) > > > @@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > > if (dev->vqs[i]->call_ctx.ctx) > > > eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); > > > vhost_vq_reset(dev, dev->vqs[i]); > > > + mutex_unlock(&dev->vqs[i]->mutex); > > > } > > > > So this is a mitigation plan but the bug is still there though > > we don't know exactly what it is. I would prefer adding something like > > WARN_ON(mutex_is_locked(vqs[i]->mutex) here - does this make sense? > > As a rework to this, or as a subsequent patch?
Can be a separate patch.
> Just before the first lock I assume?
I guess so, yes.
No problem. Patch to follow.
I'm also going to attempt to debug the root cause, but I'm new to this subsystem to it might take a while for me to get my head around.
IIUC the root cause should be the same as the one we solved here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
The worker was not stopped before calling vhost_dev_cleanup(). So while the worker was still running we were going to free memory or initialize fields while it was still using virtqueue.
Right, and I agree but it's not the root though, we do attempt to stop all workers.
Exactly. This is what happens, but the question I'm going to attempt to answer is *why* does this happen.
IIUC the worker was still running because the /dev/vhost-vsock file was not explicitly closed, so vhost_vsock_dev_release() was called in the do_exit() of the process.
In that case there was the issue, because vhost_dev_check_owner() returned false in vhost_vsock_stop() since current->mm was NULL. So it returned earlier, without calling vhost_vq_set_backend(vq, NULL).
This did not stop the worker from continuing to run, causing the multiple issues we are seeing.
current->mm was NULL, because in the do_exit() the address space is cleaned in the exit_mm(), which is called before releasing the files into the exit_task_work().
This can be seen from the logs, where we see first the warnings printed by vhost_dev_cleanup() and then the panic in the worker (e.g. here https://syzkaller.appspot.com/text?tag=CrashLog&x=16a61fce700000)
Mike also added a few more helpful details in this thread: https://lore.kernel.org/virtualization/20220221100500.2x3s2sddqahgdfyt@sgarz...
I guess that about sums it up. :)
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
So combine with the warning patch and update description with the comment I posted, explaining it's more a just in case thing.
drivers/vhost/vhost.c | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe28..bbaff6a5e21b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i; for (i = 0; i < dev->nvqs; ++i) {
mutex_lock(&dev->vqs[i]->mutex);@@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) if (dev->vqs[i]->call_ctx.ctx) eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); vhost_vq_reset(dev, dev->vqs[i]);
mutex_unlock(&dev->vqs[i]->mutex);-- 2.35.1.574.g5d30c73bfb-goog
On Fri, 04 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
So combine with the warning patch and update description with the comment I posted, explaining it's more a just in case thing.
Will do. Plan is to submit this on Monday.
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
OK so please post series with this and the warning cleaned up comments and commit logs explaining that this is just to make debugging easier in case we have issues in the future, it's not a bugfix.
drivers/vhost/vhost.c | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 59edb5a1ffe28..bbaff6a5e21b8 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -693,6 +693,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) int i; for (i = 0; i < dev->nvqs; ++i) {
mutex_lock(&dev->vqs[i]->mutex);@@ -700,6 +701,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) if (dev->vqs[i]->call_ctx.ctx) eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); vhost_vq_reset(dev, dev->vqs[i]);
mutex_unlock(&dev->vqs[i]->mutex);-- 2.35.1.574.g5d30c73bfb-goog
On Fri, 04 Mar 2022, Michael S. Tsirkin wrote:
On Wed, Mar 02, 2022 at 07:54:21AM +0000, Lee Jones wrote:
vhost_vsock_handle_tx_kick() already holds the mutex during its call to vhost_get_vq_desc(). All we have to do is take the same lock during virtqueue clean-up and we mitigate the reported issues.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: stable@vger.kernel.org Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Signed-off-by: Lee Jones lee.jones@linaro.org
OK so please post series with this and the warning cleaned up comments and commit logs explaining that this is just to make debugging easier in case we have issues in the future, it's not a bugfix.
No problem.
Just to clarify, drop Cc: Stable also?
linux-stable-mirror@lists.linaro.org
-
Lee Jones -
Michael S. Tsirkin -
Stefano Garzarella