Right now we power-up the device when a user open() the device and we power it off when the last user close() the first video node.
This behaviour affects the power consumption of the device is multiple use cases, such as: - Polling the privacy gpio - udev probing the device
This patchset introduces a more granular power saving behaviour where the camera is only awaken when needed. It is compatible with asynchronous controls.
While developing this patchset, two bugs were found. The patchset has been developed so these fixes can be taken independently.
Signed-off-by: Ricardo Ribalda ribalda@chromium.org --- Ricardo Ribalda (9): media: uvcvideo: Do not set an async control owned by other fh media: uvcvideo: Remove dangling pointers media: uvcvideo: Keep streaming state in the file handle media: uvcvideo: Move usb_autopm_(get|put)_interface to status_get media: uvcvideo: Add a uvc_status guard media: uvcvideo: Increase/decrease the PM counter per IOCTL media: uvcvideo: Make power management granular media: uvcvideo: Do not turn on the camera for some ioctls media: uvcvideo: Remove duplicated cap/out code
drivers/media/usb/uvc/uvc_ctrl.c | 52 +++++++++- drivers/media/usb/uvc/uvc_status.c | 38 +++++++- drivers/media/usb/uvc/uvc_v4l2.c | 190 +++++++++++++++---------------------- drivers/media/usb/uvc/uvcvideo.h | 6 ++ 4 files changed, 166 insertions(+), 120 deletions(-) --- base-commit: 72ad4ff638047bbbdf3232178fea4bec1f429319 change-id: 20241126-uvc-granpower-ng-069185a6d474
Best regards,
If a file handle is waiting for a response from an async control, avoid that other file handle operate with it.
Without this patch, the first file handle will never get the event associated to that operation.
Cc: stable@vger.kernel.org Fixes: e5225c820c05 ("media: uvcvideo: Send a control event when a Control Change interrupt arrives") Signed-off-by: Ricardo Ribalda ribalda@chromium.org --- drivers/media/usb/uvc/uvc_ctrl.c | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c index 4fe26e82e3d1..5d3a28edf7f0 100644 --- a/drivers/media/usb/uvc/uvc_ctrl.c +++ b/drivers/media/usb/uvc/uvc_ctrl.c @@ -1950,6 +1950,10 @@ int uvc_ctrl_set(struct uvc_fh *handle, if (!(ctrl->info.flags & UVC_CTRL_FLAG_SET_CUR)) return -EACCES;
+ /* Other file handle is waiting a response from this async control. */ + if (ctrl->handle && ctrl->handle != handle) + return -EBUSY; + /* Clamp out of range values. */ switch (mapping->v4l2_type) { case V4L2_CTRL_TYPE_INTEGER:
When an async control is written, we copy a pointer to the file handle that started the operation. That pointer will be used when the device is done. Which could be anytime in the future.
If the user closes that file descriptor, its structure will be freed, and there will be one dangling pointer per pending async control, that the driver will try to use.
Keep a counter of all the pending async controls and clean all the dangling pointers during release().
Cc: stable@vger.kernel.org Fixes: e5225c820c05 ("media: uvcvideo: Send a control event when a Control Change interrupt arrives") Signed-off-by: Ricardo Ribalda ribalda@chromium.org --- drivers/media/usb/uvc/uvc_ctrl.c | 40 ++++++++++++++++++++++++++++++++++++++-- drivers/media/usb/uvc/uvc_v4l2.c | 2 ++ drivers/media/usb/uvc/uvcvideo.h | 3 +++ 3 files changed, 43 insertions(+), 2 deletions(-)
diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c index 5d3a28edf7f0..11287e81d91c 100644 --- a/drivers/media/usb/uvc/uvc_ctrl.c +++ b/drivers/media/usb/uvc/uvc_ctrl.c @@ -1589,7 +1589,12 @@ void uvc_ctrl_status_event(struct uvc_video_chain *chain, mutex_lock(&chain->ctrl_mutex);
handle = ctrl->handle; - ctrl->handle = NULL; + if (handle) { + ctrl->handle = NULL; + WARN_ON(!handle->pending_async_ctrls) + if (handle->pending_async_ctrls) + handle->pending_async_ctrls--; + }
list_for_each_entry(mapping, &ctrl->info.mappings, list) { s32 value = __uvc_ctrl_get_value(mapping, data); @@ -2050,8 +2055,11 @@ int uvc_ctrl_set(struct uvc_fh *handle, mapping->set(mapping, value, uvc_ctrl_data(ctrl, UVC_CTRL_DATA_CURRENT));
- if (ctrl->info.flags & UVC_CTRL_FLAG_ASYNCHRONOUS) + if (ctrl->info.flags & UVC_CTRL_FLAG_ASYNCHRONOUS) { + if (!ctrl->handle) + handle->pending_async_ctrls++; ctrl->handle = handle; + }
ctrl->dirty = 1; ctrl->modified = 1; @@ -2774,6 +2782,34 @@ int uvc_ctrl_init_device(struct uvc_device *dev) return 0; }
+void uvc_ctrl_cleanup_fh(struct uvc_fh *handle) +{ + struct uvc_entity *entity; + + guard(mutex)(&handle->chain->ctrl_mutex); + + if (!handle->pending_async_ctrls) + return; + + list_for_each_entry(entity, &handle->chain->dev->entities, list) { + int i; + + for (i = 0; i < entity->ncontrols; ++i) { + struct uvc_control *ctrl = &entity->controls[i]; + + if (!ctrl->handle || ctrl->handle != handle) + continue; + + ctrl->handle = NULL; + if (WARN_ON(!handle->pending_async_ctrls)) + continue; + handle->pending_async_ctrls--; + } + } + + WARN_ON(handle->pending_async_ctrls); +} + /* * Cleanup device controls. */ diff --git a/drivers/media/usb/uvc/uvc_v4l2.c b/drivers/media/usb/uvc/uvc_v4l2.c index 97c5407f6603..b425306a3b8c 100644 --- a/drivers/media/usb/uvc/uvc_v4l2.c +++ b/drivers/media/usb/uvc/uvc_v4l2.c @@ -652,6 +652,8 @@ static int uvc_v4l2_release(struct file *file)
uvc_dbg(stream->dev, CALLS, "%s\n", __func__);
+ uvc_ctrl_cleanup_fh(handle); + /* Only free resources if this is a privileged handle. */ if (uvc_has_privileges(handle)) uvc_queue_release(&stream->queue); diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h index 07f9921d83f2..2f8a9c48e32a 100644 --- a/drivers/media/usb/uvc/uvcvideo.h +++ b/drivers/media/usb/uvc/uvcvideo.h @@ -612,6 +612,7 @@ struct uvc_fh { struct uvc_video_chain *chain; struct uvc_streaming *stream; enum uvc_handle_state state; + unsigned int pending_async_ctrls; /* Protected by ctrl_mutex. */ };
struct uvc_driver { @@ -797,6 +798,8 @@ int uvc_ctrl_is_accessible(struct uvc_video_chain *chain, u32 v4l2_id, int uvc_xu_ctrl_query(struct uvc_video_chain *chain, struct uvc_xu_control_query *xqry);
+void uvc_ctrl_cleanup_fh(struct uvc_fh *handle); + /* Utility functions */ struct usb_host_endpoint *uvc_find_endpoint(struct usb_host_interface *alts, u8 epaddr);
[Resending in plain text, seem like today is not may day]
On Tue, 26 Nov 2024 at 17:20, Ricardo Ribalda ribalda@chromium.org wrote:
When an async control is written, we copy a pointer to the file handle that started the operation. That pointer will be used when the device is done. Which could be anytime in the future.
If the user closes that file descriptor, its structure will be freed, and there will be one dangling pointer per pending async control, that the driver will try to use.
Keep a counter of all the pending async controls and clean all the dangling pointers during release().
Cc: stable@vger.kernel.org Fixes: e5225c820c05 ("media: uvcvideo: Send a control event when a Control Change interrupt arrives") Signed-off-by: Ricardo Ribalda ribalda@chromium.org
drivers/media/usb/uvc/uvc_ctrl.c | 40 ++++++++++++++++++++++++++++++++++++++-- drivers/media/usb/uvc/uvc_v4l2.c | 2 ++ drivers/media/usb/uvc/uvcvideo.h | 3 +++ 3 files changed, 43 insertions(+), 2 deletions(-)
diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c index 5d3a28edf7f0..11287e81d91c 100644 --- a/drivers/media/usb/uvc/uvc_ctrl.c +++ b/drivers/media/usb/uvc/uvc_ctrl.c @@ -1589,7 +1589,12 @@ void uvc_ctrl_status_event(struct uvc_video_chain *chain, mutex_lock(&chain->ctrl_mutex);
handle = ctrl->handle;
ctrl->handle = NULL;
if (handle) {
ctrl->handle = NULL;
WARN_ON(!handle->pending_async_ctrls)
There is obviously a missing semicolon here. I screwed it up when I reordered the patches to move it to the first part of the set. Luckily we have CI :).
You can see the fixed version here: https://gitlab.freedesktop.org/linux-media/users/ribalda/-/commits/b4/uvc-gr...
I do not plan to resend the whole series until there are more comments. But I am sending the first two patches as a new set, they are fixes. I will also send the last patch alone, it is unrelated to this.
if (handle->pending_async_ctrls)
handle->pending_async_ctrls--;
} list_for_each_entry(mapping, &ctrl->info.mappings, list) { s32 value = __uvc_ctrl_get_value(mapping, data);
@@ -2050,8 +2055,11 @@ int uvc_ctrl_set(struct uvc_fh *handle, mapping->set(mapping, value, uvc_ctrl_data(ctrl, UVC_CTRL_DATA_CURRENT));
if (ctrl->info.flags & UVC_CTRL_FLAG_ASYNCHRONOUS)
if (ctrl->info.flags & UVC_CTRL_FLAG_ASYNCHRONOUS) {
if (!ctrl->handle)
handle->pending_async_ctrls++; ctrl->handle = handle;
} ctrl->dirty = 1; ctrl->modified = 1;
@@ -2774,6 +2782,34 @@ int uvc_ctrl_init_device(struct uvc_device *dev) return 0; }
+void uvc_ctrl_cleanup_fh(struct uvc_fh *handle) +{
struct uvc_entity *entity;
guard(mutex)(&handle->chain->ctrl_mutex);
if (!handle->pending_async_ctrls)
return;
list_for_each_entry(entity, &handle->chain->dev->entities, list) {
int i;
for (i = 0; i < entity->ncontrols; ++i) {
struct uvc_control *ctrl = &entity->controls[i];
if (!ctrl->handle || ctrl->handle != handle)
continue;
ctrl->handle = NULL;
if (WARN_ON(!handle->pending_async_ctrls))
continue;
handle->pending_async_ctrls--;
}
}
WARN_ON(handle->pending_async_ctrls);
+}
/*
- Cleanup device controls.
*/ diff --git a/drivers/media/usb/uvc/uvc_v4l2.c b/drivers/media/usb/uvc/uvc_v4l2.c index 97c5407f6603..b425306a3b8c 100644 --- a/drivers/media/usb/uvc/uvc_v4l2.c +++ b/drivers/media/usb/uvc/uvc_v4l2.c @@ -652,6 +652,8 @@ static int uvc_v4l2_release(struct file *file)
uvc_dbg(stream->dev, CALLS, "%s\n", __func__);
uvc_ctrl_cleanup_fh(handle);
/* Only free resources if this is a privileged handle. */ if (uvc_has_privileges(handle)) uvc_queue_release(&stream->queue);
diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h index 07f9921d83f2..2f8a9c48e32a 100644 --- a/drivers/media/usb/uvc/uvcvideo.h +++ b/drivers/media/usb/uvc/uvcvideo.h @@ -612,6 +612,7 @@ struct uvc_fh { struct uvc_video_chain *chain; struct uvc_streaming *stream; enum uvc_handle_state state;
unsigned int pending_async_ctrls; /* Protected by ctrl_mutex. */
};
struct uvc_driver { @@ -797,6 +798,8 @@ int uvc_ctrl_is_accessible(struct uvc_video_chain *chain, u32 v4l2_id, int uvc_xu_ctrl_query(struct uvc_video_chain *chain, struct uvc_xu_control_query *xqry);
+void uvc_ctrl_cleanup_fh(struct uvc_fh *handle);
/* Utility functions */ struct usb_host_endpoint *uvc_find_endpoint(struct usb_host_interface *alts, u8 epaddr);
-- 2.47.0.338.g60cca15819-goog
Hi Ricardo,
On 26-Nov-24 5:18 PM, Ricardo Ribalda wrote:
Right now we power-up the device when a user open() the device and we power it off when the last user close() the first video node.
This behaviour affects the power consumption of the device is multiple use cases, such as:
- Polling the privacy gpio
- udev probing the device
This patchset introduces a more granular power saving behaviour where the camera is only awaken when needed. It is compatible with asynchronous controls.
While developing this patchset, two bugs were found. The patchset has been developed so these fixes can be taken independently.
Thank you for your patch series. For now lets focus on fixing the async-controls ctrl->handle setting / dangling ptr issue and then we can look into the rest of this later (after we have also landed the privacy GPIO and UVC 1.5 ROi series).
Regards,
Hans
Ricardo Ribalda (9): media: uvcvideo: Do not set an async control owned by other fh media: uvcvideo: Remove dangling pointers media: uvcvideo: Keep streaming state in the file handle media: uvcvideo: Move usb_autopm_(get|put)_interface to status_get media: uvcvideo: Add a uvc_status guard media: uvcvideo: Increase/decrease the PM counter per IOCTL media: uvcvideo: Make power management granular media: uvcvideo: Do not turn on the camera for some ioctls media: uvcvideo: Remove duplicated cap/out code
drivers/media/usb/uvc/uvc_ctrl.c | 52 +++++++++- drivers/media/usb/uvc/uvc_status.c | 38 +++++++- drivers/media/usb/uvc/uvc_v4l2.c | 190 +++++++++++++++---------------------- drivers/media/usb/uvc/uvcvideo.h | 6 ++ 4 files changed, 166 insertions(+), 120 deletions(-)
base-commit: 72ad4ff638047bbbdf3232178fea4bec1f429319 change-id: 20241126-uvc-granpower-ng-069185a6d474
Best regards,
linux-stable-mirror@lists.linaro.org