Hi,
Thanks for your ideas.
If I am not mistaken all subdevices in the ISP media pipe could be interconnected without the need from ARM intervention. But I could be wrong.
Why not ask Hans Verkuil and Laurent Pinchart? Sorry for bringing you in like this but you are the true experts in v4l2. Do you guys have any comments on the discussions in this mail?
/BR /Robert Fekete
On 8 February 2011 13:42, SUBASH PATEL subash.rp@samsung.com wrote:
Hi Robert,
Thanks for sharing the slide. It was informative on OMAP3. My reference was wrt to a similar hardware itself.
In slide 20, the green blobs are mentioned to be drivers. Lets consider an example where camera sensor is connected on a CSI2 interface.
On every CSI2 interrupt (assuming frame based), we will have to take out frame and pass on to each of ISP block in the diagram. ARM will
be involved.
If we are speaking of performance like HD/Full-HD @ 30fps, imagine the processing required by the ARM processor for this. Everytime it has to
take frame from one component and pass onto another until resizer gives a final frame. From there it has to go to an encoder for encoding too.
But if we use a dedicated imaging processor, which will run on its own and provide us the desired frames (last yellow box - ISP resizer
output), ARM can concentrate on something else in the meantime.
We cannot do such a thing with V4L2. As far my knowledge is, since this is two processor environment, we require some client/server architecture. OMX
comes handy in these cases as it has all of client, core and component parts. Thats why gstreamer is looked as a broker by many media applications.
Gstreamer will appropriately forward controls to v4L2 or OMX depending on how the hardware is delivering the frames.
Regards,
Subash
------- *Original Message* -------
*Sender* : Robert Feketerobert.fekete@linaro.org
*Date* : Feb 08, 2011 17:40 (GMT+05:30)
*Title* : Re: v4l2 vs omx for camera
Hi,
This presentation from Hans Verkaul last V4L2 summit describes the Media controller which is s perfect fit for an ISP. Pay special attention to Slide 20. Yellow boxes are input/outpud devices, green blobs are subdevices/drivers
Thus V4L2 will fit any camera sensor whether it is yuv or raw camera. also providing a neat kernel interface with all source available for customers and happy hackers to use.
BR /Robert F
On 8 February 2011 11:42, SUBASH PATEL subash.rp@samsung.com wrote:
Hi Sachin,
I think when we speak of OMX, we are referring to the OMX-IL layer. This layer is supported as middleware component.
I am putting down my experiences as below:
- Generally Camera gives two streams. One is preview which can be YUV/RGB
and another is capture (YUV/RGB/JPEG). Preview frame format must be in one of display systems supported(YUV/RGB) format. Else color conversion is required in the path. This adds overhead and latency.
- If we have a camera sensor which is smart, i.e., it is capable of
providing the processed image(RGB, YUV) frame rather than RAW pixel dump, and ARM is able to control the sensor interface, then V4L2 framework camera driver would work. User space wrapper/app would be invoking the V4L2 ioctl's to control the camera.
- If we have a RAW sensor which produces, say Bayer pixel format, we will
have to have an image pipe to process it before converting this to one of RGB/YUV formats. Image pipes involves conversions, resizing etc. It would be an overhead to do these stages in ARM, and some vendors have proprietary imaging processors for it. These processors may run a custom RTOS.
They may have built a private IPC layer into linux kernel and proprietary OS. OMX layer works in such scenarios. A concept called distributed OMX works on RPC mechanism. OMX client calls will now land up on the image processor from ARM. Again some proprietary driver/s will be invoked from the remote OMX component which would do the image processing mentioned above.
User space wrapper/app aka OMX client, is a set of OMX calls, which will get routed to proper OMX component through OMX core. This is similar to V4L2 client, but instead of controlling camera through IOCTL's, we use OMX specific functions Get/Set methods.
From my view, the choice of choosing V4L2 or OMX is basically depending on the type of sensors and presence of dedicated hardware. If we already have a dedicated imaging processor, V4L2 can be absent, and we will have to leverage the OMX because of its capability. But if we are integrating a new sensor which has in-built accelerator, it makes sense to reduce the silicon area on SoC and use V4L2 instead.
Regards, Subash -------Original Message-------- Sent: Sachin Gupta sachin.gupta@linaro.org Date: Tue, 8 Feb 2011 14:25:21 +0530 Subject: Re: v4l2 vs omx for camera
Arnd,
you are correct that omx and v4l2 sit at different levels one being userside API and other being kernel API.But from the point of view of integrating these API's in OS frameworks like gstreamer,Android camera service they are at the same level.I mean one will have to implement gstreamer source plugin based on either v4l2 or Omx.Also the way vendors(STE and TI) have gone about implementing OMX, they completely bypass v4l2 .The major reason being code sharing among different OS environments.The kernel side for OMX implementation just facilitates RPC between imaging coprocessor and ARM side..
Sachin
On Tue, Feb 8, 2011 at 2:00 PM, Lee Jones lee.jones@linaro.org wrote:
Bringing in my boys.
Robert, Linus, what say you?
On 07/02/11 12:33, Arnd Bergmann wrote:
On Monday 07 February 2011, Sachin Gupta wrote:
In Multimedia WG we have been posed with a question regarding bestway
to expose low level API for camera.so this a questions mainly about
pros and
cons of v4l2 and omx over each other.So to involve a wider community to discuss this topic I am floating this mail on linaro-dev.Please share
your
view/experiences.Also please involve any body else in this mail who can provide valuable inputs on this.
I've had to look up with "omx" actually stands for [1][2], but from an outsider view, they don't seem to be mutually exclusive or even competing interfaces. v4l2 is the interface you use to get at camera data, in whatever format the camera gives you. There are no alternatives to that. OpenMax gives you a way to accelerate video codecs, which is good, but this sits a layer higher up in the stack. Supporting omx is probably a good idea, but would be totally optional.
Arnd[1] http://www.khronos.org/openmax/ [2] http://www.freedesktop.org/wiki/GstOpenMAX
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
-------Original Message-------- Sent: Loïc Minier loic.minier@linaro.org Date: Tue, 8 Feb 2011 10:35:38 +0100 Subject: Re: Efikamx bootloader help
hey adding my bits where I can On Tue, Feb 08, 2011, Eric Miao wrote: > 2. The three possible boot up methods: a) Internal SPI NOR flash, b) the > MicroSD card behind the battery and c) the normal SD card at the left > side, not really sure about the situation on Efika MX (smart top) as > I don't have one in my hands efikamx only has SD and internal flash; there might be other boot methods like serial or USB, but I don't think we need to care too much about these > 4. The boot sequence of the Internal SPI NOR flash, as it looks to me that > it's trying to find boot.scr from either the MicroSD or SD card on the > first partition? Also it would be helpful to document the envionment > variables of this specific u-boot. on my efikamx, this is the bootcmd it had when I received it and corresponding variables: bootcmd=run pata_boot pata_boot=run bootargs_base bootargs_pata bootargs_base=setenv bootargs noinitrd console=ttymxc0,115200 console=tty1 bootargs_pata=setenv bootargs ${bootargs} root=/dev/sda2 ${bootinfo};run boot_pata bootinfo=rw boot_pata=run base_cmds;ide reset;fatload ide 0:1 ${loadaddr} ${kernel}; bootm ${loadaddr} base_cmds=run base_cmd1;run base_cmd2;run base_cmd3;run base_cmd4;run base_cmd5;run base_cmd99 loadaddr=0x90007FC0 kernel=uImage base_cmd1=pmic 15 0x00400022;mw.l 0x73fa84b8 0xe7 1;mw.l 0x73fd4014 0x59239100 1 base_cmd2=mw.l 0x83fd9010 0xcaaaf6d0 1;mw.l 0x73f88000 0x01025200 1;mw.l 0x73f84000 0x20 1 base_cmd3=mw.l 0x83fd9004 0x333574aa 1;mw.l 0x83fd900c 0x333574aa 1; base_cmd4=mw.l 0x83fd9020 0x00f48b00 1;mw.l 0x83fd9024 0x00f49700 1;mw.l 0x83fd9028 0x00f48700 1 base_cmd5=mw.l 0x83fd902c 0x00f48400 1;mw.l 0x83fd9030 0x00f44e00 1; > 5. If Linaro is going to support u-boot for Efika MX/SB, it's better to > follow what is in upstream. However, there could be some differences > between the u-boot in the recovery/installing image downloadable from > powerdeveloper.org and the one in upstream. We might need to figure > out those differences and see how to handle them. That was indeed the case; Marex, who developed the upstream u-boot bits, told me the same u-boot would work on SD and on flash; the one I've built from mainline uses the default imximage.cfg settings which is "BOOT_FROM spi", yet works fine from a SD card. So it seems the same u-boot can be used for both SD and flash -- just with different bootcmd. Cheers, -- Loïc Minier _______________________________________________ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Hi,
Thanks for your ideas.
If I am not mistaken all subdevices in the ISP media pipe could be interconnected without the need from ARM intervention. But I could be wrong.
You are completely right. V4L2 sets up the internal hardware pipeline. Once you start streaming it is all done in hardware. It would be sad indeed if you would have to pass the frames around manually.
OMX tends to be geared towards DSPs (encoders/decoders). In theory it would be possible to use V4L2 for this as well, but nobody has done so. Generally pure hardware encoders/decoders *are* handled by V4L2 (there are a few drivers that do that).
Linux implementations of OMX should really use V4L2 to control the underlying video hardware. But due to the often close-source nature of codecs and due to the limited functionality of the V4L2 API (at least up to a few years ago) this was probably never a consideration.
However, the V4L2 API and internal framework is improving rapidly, so it is now much more attractive for powerful video hardware.
BTW, the main mailinglist for all things V4L is linux-media@vger.kernel.org.
Regards,
Hans
Why not ask Hans Verkuil and Laurent Pinchart? Sorry for bringing you in like this but you are the true experts in v4l2. Do you guys have any comments on the discussions in this mail?
/BR /Robert Fekete
On 8 February 2011 13:42, SUBASH PATEL subash.rp@samsung.com wrote:
Hi Robert,
Thanks for sharing the slide. It was informative on OMAP3. My reference was wrt to a similar hardware itself.
In slide 20, the green blobs are mentioned to be drivers. Lets consider an example where camera sensor is connected on a CSI2 interface.
On every CSI2 interrupt (assuming frame based), we will have to take out frame and pass on to each of ISP block in the diagram. ARM will
be involved.
If we are speaking of performance like HD/Full-HD @ 30fps, imagine the processing required by the ARM processor for this. Everytime it has to
take frame from one component and pass onto another until resizer gives a final frame. From there it has to go to an encoder for encoding too.
But if we use a dedicated imaging processor, which will run on its own and provide us the desired frames (last yellow box - ISP resizer
output), ARM can concentrate on something else in the meantime.
We cannot do such a thing with V4L2. As far my knowledge is, since this is two processor environment, we require some client/server architecture. OMX
comes handy in these cases as it has all of client, core and component parts. Thats why gstreamer is looked as a broker by many media applications.
Gstreamer will appropriately forward controls to v4L2 or OMX depending on how the hardware is delivering the frames.
Regards,
Subash
------- *Original Message* -------
*Sender* : Robert Feketerobert.fekete@linaro.org
*Date* : Feb 08, 2011 17:40 (GMT+05:30)
*Title* : Re: v4l2 vs omx for camera
Hi,
This presentation from Hans Verkaul last V4L2 summit describes the Media controller which is s perfect fit for an ISP. Pay special attention to Slide 20. Yellow boxes are input/outpud devices, green blobs are subdevices/drivers
Thus V4L2 will fit any camera sensor whether it is yuv or raw camera. also providing a neat kernel interface with all source available for customers and happy hackers to use.
BR /Robert F
On 8 February 2011 11:42, SUBASH PATEL subash.rp@samsung.com wrote:
Hi Sachin,
I think when we speak of OMX, we are referring to the OMX-IL layer. This layer is supported as middleware component.
I am putting down my experiences as below:
- Generally Camera gives two streams. One is preview which can be
YUV/RGB and another is capture (YUV/RGB/JPEG). Preview frame format must be in one of display systems supported(YUV/RGB) format. Else color conversion is required in the path. This adds overhead and latency.
- If we have a camera sensor which is smart, i.e., it is capable of
providing the processed image(RGB, YUV) frame rather than RAW pixel dump, and ARM is able to control the sensor interface, then V4L2 framework camera driver would work. User space wrapper/app would be invoking the V4L2 ioctl's to control the camera.
- If we have a RAW sensor which produces, say Bayer pixel format, we
will have to have an image pipe to process it before converting this to one of RGB/YUV formats. Image pipes involves conversions, resizing etc. It would be an overhead to do these stages in ARM, and some vendors have proprietary imaging processors for it. These processors may run a custom RTOS.
They may have built a private IPC layer into linux kernel and proprietary OS. OMX layer works in such scenarios. A concept called distributed OMX works on RPC mechanism. OMX client calls will now land up on the image processor from ARM. Again some proprietary driver/s will be invoked from the remote OMX component which would do the image processing mentioned above.
User space wrapper/app aka OMX client, is a set of OMX calls, which will get routed to proper OMX component through OMX core. This is similar to V4L2 client, but instead of controlling camera through IOCTL's, we use OMX specific functions Get/Set methods.
From my view, the choice of choosing V4L2 or OMX is basically depending on the type of sensors and presence of dedicated hardware. If we already have a dedicated imaging processor, V4L2 can be absent, and we will have to leverage the OMX because of its capability. But if we are integrating a new sensor which has in-built accelerator, it makes sense to reduce the silicon area on SoC and use V4L2 instead.
Regards, Subash -------Original Message-------- Sent: Sachin Gupta sachin.gupta@linaro.org Date: Tue, 8 Feb 2011 14:25:21 +0530 Subject: Re: v4l2 vs omx for camera
Arnd,
you are correct that omx and v4l2 sit at different levels one being userside API and other being kernel API.But from the point of view of integrating these API's in OS frameworks like gstreamer,Android camera service they are at the same level.I mean one will have to implement gstreamer source plugin based on either v4l2 or Omx.Also the way vendors(STE and TI) have gone about implementing OMX, they completely bypass v4l2 .The major reason being code sharing among different OS environments.The kernel side for OMX implementation just facilitates RPC between imaging coprocessor and ARM side..
Sachin
On Tue, Feb 8, 2011 at 2:00 PM, Lee Jones lee.jones@linaro.org wrote:
Bringing in my boys.
Robert, Linus, what say you?
On 07/02/11 12:33, Arnd Bergmann wrote:
On Monday 07 February 2011, Sachin Gupta wrote:
In Multimedia WG we have been posed with a question regardingbest way
to expose low level API for camera.so this a questions mainly about
pros and
cons of v4l2 and omx over each other.So to involve a wider community
to
discuss this topic I am floating this mail on linaro-dev.Please
share your
view/experiences.Also please involve any body else in this mail who
can
provide valuable inputs on this.
I've had to look up with "omx" actually stands for [1][2], but from an outsider view, they don't seem to be mutually exclusive or even competing interfaces. v4l2 is the interface you use to get at camera data, in whatever format the camera gives you. There are no
alternatives
to that. OpenMax gives you a way to accelerate video codecs, which is good, but this sits a layer higher up in the stack. Supporting omx is probably a good idea, but would be totally optional.
Arnd[1] http://www.khronos.org/openmax/ [2] http://www.freedesktop.org/wiki/GstOpenMAX
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
-------Original Message-------- Sent: Loïc Minier loic.minier@linaro.org Date: Tue, 8 Feb 2011 10:35:38 +0100 Subject: Re: Efikamx bootloader help
hey adding my bits where I can On Tue, Feb 08, 2011, Eric Miao wrote: > 2. The three possible boot up methods: a) Internal SPI NOR flash, b) the > MicroSD card behind the battery and c) the normal SD card at the left > side, not really sure about the situation on Efika MX (smart top) as > I don't have one in my hands efikamx only has SD and internal flash; there might be other boot methods like serial or USB, but I don't think we need to care too much about these > 4. The boot sequence of the Internal SPI NOR flash, as it looks to me that > it's trying to find boot.scr from either the MicroSD or SD card on the > first partition? Also it would be helpful to document the envionment > variables of this specific u-boot. on my efikamx, this is the bootcmd it had when I received it and corresponding variables: bootcmd=run pata_boot pata_boot=run bootargs_base bootargs_pata bootargs_base=setenv bootargs noinitrd console=ttymxc0,115200 console=tty1 bootargs_pata=setenv bootargs ${bootargs} root=/dev/sda2 ${bootinfo};run boot_pata bootinfo=rw boot_pata=run base_cmds;ide reset;fatload ide 0:1 ${loadaddr} ${kernel}; bootm ${loadaddr} base_cmds=run base_cmd1;run base_cmd2;run base_cmd3;run base_cmd4;run base_cmd5;run base_cmd99 loadaddr=0x90007FC0 kernel=uImage base_cmd1=pmic 15 0x00400022;mw.l 0x73fa84b8 0xe7 1;mw.l 0x73fd4014 0x59239100 1 base_cmd2=mw.l 0x83fd9010 0xcaaaf6d0 1;mw.l 0x73f88000 0x01025200 1;mw.l 0x73f84000 0x20 1 base_cmd3=mw.l 0x83fd9004 0x333574aa 1;mw.l 0x83fd900c 0x333574aa 1; base_cmd4=mw.l 0x83fd9020 0x00f48b00 1;mw.l 0x83fd9024 0x00f49700 1;mw.l 0x83fd9028 0x00f48700 1 base_cmd5=mw.l 0x83fd902c 0x00f48400 1;mw.l 0x83fd9030 0x00f44e00 1; > 5. If Linaro is going to support u-boot for Efika MX/SB, it's better to > follow what is in upstream. However, there could be some differences > between the u-boot in the recovery/installing image downloadable from > powerdeveloper.org and the one in upstream. We might need to figure > out those differences and see how to handle them. That was indeed the case; Marex, who developed the upstream u-boot bits, told me the same u-boot would work on SD and on flash; the one I've built from mainline uses the default imximage.cfg settings which is "BOOT_FROM spi", yet works fine from a SD card. So it seems the same u-boot can be used for both SD and flash -- just with different bootcmd. Cheers, -- Loïc Minier _______________________________________________ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Hi Robert and all,
On Tuesday 08 February 2011 14:48:21 Robert Fekete wrote:
Hi,
Thanks for your ideas.
If I am not mistaken all subdevices in the ISP media pipe could be interconnected without the need from ARM intervention. But I could be wrong.
Why not ask Hans Verkuil and Laurent Pinchart? Sorry for bringing you in like this but you are the true experts in v4l2.
No worries, I'm glad to be helpful.
On 8 February 2011 13:42, SUBASH PATEL subash.rp@samsung.com wrote:
Thanks for sharing the slide. It was informative on OMAP3. My reference was wrt to a similar hardware itself.
In slide 20, the green blobs are mentioned to be drivers. Lets consider an example where camera sensor is connected on a CSI2 interface.
On every CSI2 interrupt (assuming frame based), we will have to take out frame and pass on to each of ISP block in the diagram. ARM will be involved.
If we are speaking of performance like HD/Full-HD @ 30fps, imagine the processing required by the ARM processor for this. Everytime it has to take frame from one component and pass onto another until resizer gives a final frame. From there it has to go to an encoder for encoding too.
Whether the ARM processor is involved largely depends on the hardware. The OMAP3 ISP can process video frames from the CSI2 block to the resizer without going through memory and without generating interrupts for intermediate results. The ARM processor will only be interrupted when the resizer has finished processing the frame.
But if we use a dedicated imaging processor, which will run on its own and provide us the desired frames (last yellow box - ISP resizer output), ARM can concentrate on something else in the meantime.
We cannot do such a thing with V4L2. As far my knowledge is, since this is two processor environment, we require some client/server architecture.
There's some misunderstanding here. V4L2 is a kernel <-> userspace API used to configure the image capture device and manage image buffers. Whether the hardware is directly mapped in the ARM processor memory space, or requires communication with another processor, is an implementation detail.
OMX comes handy in these cases as it has all of client, core and component parts. Thats why gstreamer is looked as a broker by many media applications.
Gstreamer will appropriately forward controls to v4L2 or OMX depending on how the hardware is delivering the frames.
Nothing prevents you from writing a V4L2 driver that will communicate with a second CPU using an IPC mechanism. Once again V4L2 is just an API, and with the recent additions (such as the media controller API that will be merged in 2.6.39) you should be able to control any kind of hardware.
Do you have any OMX-based hardware in mind for which you think the V4L2 and media controller APIs couldn't be used efficiently ?
-
Hans Verkuil -
Laurent Pinchart -
Robert Fekete