| Message ID | 20210429132833.2802390-1-stanimir.varbanov@linaro.org |
|---|---|
| Headers | show |
| Series | HEIC image encoder | expand |
Le jeudi 29 avril 2021 à 16:28 +0300, Stanimir Varbanov a écrit : > Hi, > > HEIC (High-Efficiency Image Container) is a variant of HEIF (High > Efficiency Image File Format) where HEVC/h265 codec is used to encode > images. For more info see [1]. > > In this RFC we propose a new compressed pixel format V4L2_PIX_FMT_HEIC. > The name is debatable and could be changed (V4L2_PIX_FMT_HEVC_HEIF is > also an option). > > There are two encoding modes which should be selectable by clients: > 1. Regular image encoding > 2. Grid image encoding > > 1. Regular image encoding > > Propose to reuse stateful video encoder spec [2]. > > - queuing one OUTPUT buffer will produce one CAPTURE buffer. The > client could trigger Drain sequence at any point of time. Do you know the rationale why the normal HEVC encoder isn't used and then muxed into the HEIF container ? It is these days quite atypical for HW to handle the container part. Perhaps they hacked the header, I am not so familiar with these new ISOMBF based image format (AV1 got something very similar, though less convoluted). > > 2. Grid image encoding > > Propose to reuse stateful video encoder spec [2]. > > - queuing one OUTPUT buffer will produce a number of grids CAPTURE > buffers. The client could trigger Drain sequence at any point of time. > > This image encoding mode is used when the input image resolution is > bigger then the hardware can support and/or for compatibility reasons > (for exmaple, the HEIC decoding hardware is not capable to decode higher > than VGA resolutions). > > In this mode the input image is divided on square blocks (we call them grids) > and every block is encoded separately (the Venus driver presently supports > grid size of 512x512 but that could be changed in the future). Are the blocks always power of two ? Or multiple of 16 ? How do you expose this information to application ? It sounds like an HW decoder would also need to expose these restrictions. Userspace will need to be able to check without trying if the HW decoder handles the grid side to be able to fallback to software decoding. > > To set the grid size we use a new v4l2 control. > > The side effect of this mode is that the client have to set the v4l2 > control and thus enable grid encoding before setting the formats on > CAPTURE and OUTPUT queues, because the grid size reflects on the > selected resolutions. Also the horizontal and vertical strides will > also be affected because thеy have to be aligned to the grid size > in order to satisfy DMA alignment restrictions. > > Using of v4l2 control to set up Grid mode and Grid size above looks > inpractical and somehow breaks the v4l2 and v4l2 control rules, so > I'd give one more option. The stasteless decoders uses a control that must be set after the OUTPUT format, but before enumerating capture formats. Not exactly the same, but there is a control that interact with the format negotiation. > > Encoding the Grid mode/size in the new proposed HEIC pixel format: > > V4L2_PIX_FMT_HEIC - Regular HEIC image encoding > V4L2_PIX_FMT_HEIC_GRID_512x512 - Grid HEIC image encoding, grid size of 512x512 > and so on ... I would be worry of the about of formats that may generates. > > Comments and suggestions are welcome! > > regards, > Stan > > [1] https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format > [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/dev-encoder.html > > > Stanimir Varbanov (4): > media: Add HEIC compressed pixel format > v4l2-ctrls: Add HEIC grid size control > venus: helpers: Add a new helper for buffer processing > venus: Add HEIC image encoder > > .../media/v4l/pixfmt-compressed.rst | 12 + > drivers/media/platform/qcom/venus/Makefile | 2 + > drivers/media/platform/qcom/venus/core.h | 10 + > drivers/media/platform/qcom/venus/helpers.c | 20 + > drivers/media/platform/qcom/venus/helpers.h | 1 + > drivers/media/platform/qcom/venus/hfi_cmds.c | 10 +- > .../media/platform/qcom/venus/hfi_helper.h | 5 + > drivers/media/platform/qcom/venus/ienc.c | 1348 +++++++++++++++++ > drivers/media/platform/qcom/venus/ienc.h | 14 + > .../media/platform/qcom/venus/ienc_ctrls.c | 83 + > drivers/media/v4l2-core/v4l2-ctrls.c | 3 + > drivers/media/v4l2-core/v4l2-ioctl.c | 1 + > include/uapi/linux/v4l2-controls.h | 1 + > include/uapi/linux/videodev2.h | 1 + > 14 files changed, 1510 insertions(+), 1 deletion(-) > create mode 100644 drivers/media/platform/qcom/venus/ienc.c > create mode 100644 drivers/media/platform/qcom/venus/ienc.h > create mode 100644 drivers/media/platform/qcom/venus/ienc_ctrls.c >
Hi Nicolas, Thanks for comments! On 5/18/21 8:07 PM, Nicolas Dufresne wrote: > Le jeudi 29 avril 2021 à 16:28 +0300, Stanimir Varbanov a écrit : >> Hi, >> >> HEIC (High-Efficiency Image Container) is a variant of HEIF (High >> Efficiency Image File Format) where HEVC/h265 codec is used to encode >> images. For more info see [1]. >> >> In this RFC we propose a new compressed pixel format V4L2_PIX_FMT_HEIC. >> The name is debatable and could be changed (V4L2_PIX_FMT_HEVC_HEIF is >> also an option). >> >> There are two encoding modes which should be selectable by clients: >> 1. Regular image encoding >> 2. Grid image encoding >> >> 1. Regular image encoding >> >> Propose to reuse stateful video encoder spec [2]. >> >> - queuing one OUTPUT buffer will produce one CAPTURE buffer. The >> client could trigger Drain sequence at any point of time. > > Do you know the rationale why the normal HEVC encoder isn't used and then muxed > into the HEIF container ? It is these days quite atypical for HW to handle the > container part. Perhaps they hacked the header, I am not so familiar with these > new ISOMBF based image format (AV1 got something very similar, though less > convoluted). Maybe I did not explain well, but the Venus (and all the knowledge I have is based on it) does not implement the container part. The container part is implemented in client. For example I used libheif to create .heic image file from Venus encoded hevc bitstream. >> >> 2. Grid image encoding >> >> Propose to reuse stateful video encoder spec [2]. >> >> - queuing one OUTPUT buffer will produce a number of grids CAPTURE >> buffers. The client could trigger Drain sequence at any point of time. >> >> This image encoding mode is used when the input image resolution is >> bigger then the hardware can support and/or for compatibility reasons >> (for exmaple, the HEIC decoding hardware is not capable to decode higher >> than VGA resolutions). >> >> In this mode the input image is divided on square blocks (we call them grids) >> and every block is encoded separately (the Venus driver presently supports >> grid size of 512x512 but that could be changed in the future). > > Are the blocks always power of two ? Or multiple of 16 ? How do you expose this I guess the blocks will definitely be a power of two. As far as for multiple of 16, I guess grid size should be multiple of CTUs (32x32 or 64x64) in case of HEVC. It might be different for the other codecs. > information to application ? It sounds like an HW decoder would also need to > expose these restrictions. Userspace will need to be able to check without > trying if the HW decoder handles the grid side to be able to fallback to > software decoding. Depending on what we will decide : - use v4l2 control for setting the grid size then we can use VIDIOC_QUERYCTRL - if we decide to set the grid size on the CAPTURE queue SFMT the we can use VIDIOC_ENUM_FMT > >> >> To set the grid size we use a new v4l2 control. >> >> The side effect of this mode is that the client have to set the v4l2 >> control and thus enable grid encoding before setting the formats on >> CAPTURE and OUTPUT queues, because the grid size reflects on the >> selected resolutions. Also the horizontal and vertical strides will >> also be affected because thеy have to be aligned to the grid size >> in order to satisfy DMA alignment restrictions. >> >> Using of v4l2 control to set up Grid mode and Grid size above looks >> inpractical and somehow breaks the v4l2 and v4l2 control rules, so >> I'd give one more option. > > The stasteless decoders uses a control that must be set after the OUTPUT format, > but before enumerating capture formats. Not exactly the same, but there is a > control that interact with the format negotiation. I'd try to avoid such control if possible. > >> >> Encoding the Grid mode/size in the new proposed HEIC pixel format: >> >> V4L2_PIX_FMT_HEIC - Regular HEIC image encoding >> V4L2_PIX_FMT_HEIC_GRID_512x512 - Grid HEIC image encoding, grid size of 512x512 >> and so on ... > > I would be worry of the about of formats that may generates. > >> >> Comments and suggestions are welcome! >> >> regards, >> Stan >> >> [1] https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format >> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/dev-encoder.html >> >> >> Stanimir Varbanov (4): >> media: Add HEIC compressed pixel format >> v4l2-ctrls: Add HEIC grid size control >> venus: helpers: Add a new helper for buffer processing >> venus: Add HEIC image encoder >> >> .../media/v4l/pixfmt-compressed.rst | 12 + >> drivers/media/platform/qcom/venus/Makefile | 2 + >> drivers/media/platform/qcom/venus/core.h | 10 + >> drivers/media/platform/qcom/venus/helpers.c | 20 + >> drivers/media/platform/qcom/venus/helpers.h | 1 + >> drivers/media/platform/qcom/venus/hfi_cmds.c | 10 +- >> .../media/platform/qcom/venus/hfi_helper.h | 5 + >> drivers/media/platform/qcom/venus/ienc.c | 1348 +++++++++++++++++ >> drivers/media/platform/qcom/venus/ienc.h | 14 + >> .../media/platform/qcom/venus/ienc_ctrls.c | 83 + >> drivers/media/v4l2-core/v4l2-ctrls.c | 3 + >> drivers/media/v4l2-core/v4l2-ioctl.c | 1 + >> include/uapi/linux/v4l2-controls.h | 1 + >> include/uapi/linux/videodev2.h | 1 + >> 14 files changed, 1510 insertions(+), 1 deletion(-) >> create mode 100644 drivers/media/platform/qcom/venus/ienc.c >> create mode 100644 drivers/media/platform/qcom/venus/ienc.h >> create mode 100644 drivers/media/platform/qcom/venus/ienc_ctrls.c >> > > -- regards, Stan
Hi Stanimir, On 29/04/2021 15:28, Stanimir Varbanov wrote: > Hi, > > HEIC (High-Efficiency Image Container) is a variant of HEIF (High > Efficiency Image File Format) where HEVC/h265 codec is used to encode > images. For more info see [1]. > > In this RFC we propose a new compressed pixel format V4L2_PIX_FMT_HEIC. > The name is debatable and could be changed (V4L2_PIX_FMT_HEVC_HEIF is > also an option). > > There are two encoding modes which should be selectable by clients: > 1. Regular image encoding > 2. Grid image encoding > > 1. Regular image encoding > > Propose to reuse stateful video encoder spec [2]. > > - queuing one OUTPUT buffer will produce one CAPTURE buffer. The > client could trigger Drain sequence at any point of time. > > 2. Grid image encoding > > Propose to reuse stateful video encoder spec [2]. > > - queuing one OUTPUT buffer will produce a number of grids CAPTURE > buffers. The client could trigger Drain sequence at any point of time. > > This image encoding mode is used when the input image resolution is > bigger then the hardware can support and/or for compatibility reasons > (for exmaple, the HEIC decoding hardware is not capable to decode higher > than VGA resolutions). Is grid image encoding part of the spec for this format? Is this something that the venus hardware needs due to image resolution limitations as described above? Would it be possible for the driver to handle this internally? I.e., if it detects that it needs to switch to grid mode, can it just encode each grid and copy it in the capture buffer? This assumes that there is metadata that can be used by a decoder do find and decode each grid. > > In this mode the input image is divided on square blocks (we call them grids) > and every block is encoded separately (the Venus driver presently supports > grid size of 512x512 but that could be changed in the future). > > To set the grid size we use a new v4l2 control. Can the driver make a choice of the grid size, and the control just reports the grid size? I.e., does it make sense for userspace to set this? The wiki page [1] doesn't mention grids, so where does this come from? Is it part of some spec? Or is it a venus-specific feature? > > The side effect of this mode is that the client have to set the v4l2 > control and thus enable grid encoding before setting the formats on > CAPTURE and OUTPUT queues, because the grid size reflects on the > selected resolutions. Also the horizontal and vertical strides will > also be affected because thеy have to be aligned to the grid size > in order to satisfy DMA alignment restrictions. > > Using of v4l2 control to set up Grid mode and Grid size above looks > inpractical and somehow breaks the v4l2 and v4l2 control rules, so > I'd give one more option. > > Encoding the Grid mode/size in the new proposed HEIC pixel format: > > V4L2_PIX_FMT_HEIC - Regular HEIC image encoding > V4L2_PIX_FMT_HEIC_GRID_512x512 - Grid HEIC image encoding, grid size of 512x512 > and so on ... > > Comments and suggestions are welcome! I notice that this RFC just talks about the encoder, does venus also support a decoder? How would a HW decoder handle grids? Regards, Hans > > regards, > Stan > > [1] https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format > [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/dev-encoder.html > > > Stanimir Varbanov (4): > media: Add HEIC compressed pixel format > v4l2-ctrls: Add HEIC grid size control > venus: helpers: Add a new helper for buffer processing > venus: Add HEIC image encoder > > .../media/v4l/pixfmt-compressed.rst | 12 + > drivers/media/platform/qcom/venus/Makefile | 2 + > drivers/media/platform/qcom/venus/core.h | 10 + > drivers/media/platform/qcom/venus/helpers.c | 20 + > drivers/media/platform/qcom/venus/helpers.h | 1 + > drivers/media/platform/qcom/venus/hfi_cmds.c | 10 +- > .../media/platform/qcom/venus/hfi_helper.h | 5 + > drivers/media/platform/qcom/venus/ienc.c | 1348 +++++++++++++++++ > drivers/media/platform/qcom/venus/ienc.h | 14 + > .../media/platform/qcom/venus/ienc_ctrls.c | 83 + > drivers/media/v4l2-core/v4l2-ctrls.c | 3 + > drivers/media/v4l2-core/v4l2-ioctl.c | 1 + > include/uapi/linux/v4l2-controls.h | 1 + > include/uapi/linux/videodev2.h | 1 + > 14 files changed, 1510 insertions(+), 1 deletion(-) > create mode 100644 drivers/media/platform/qcom/venus/ienc.c > create mode 100644 drivers/media/platform/qcom/venus/ienc.h > create mode 100644 drivers/media/platform/qcom/venus/ienc_ctrls.c >
Hi Hans, On 5/27/21 10:54 AM, Hans Verkuil wrote: > Hi Stanimir, > > On 29/04/2021 15:28, Stanimir Varbanov wrote: >> Hi, >> >> HEIC (High-Efficiency Image Container) is a variant of HEIF (High >> Efficiency Image File Format) where HEVC/h265 codec is used to encode >> images. For more info see [1]. >> >> In this RFC we propose a new compressed pixel format V4L2_PIX_FMT_HEIC. >> The name is debatable and could be changed (V4L2_PIX_FMT_HEVC_HEIF is >> also an option). >> >> There are two encoding modes which should be selectable by clients: >> 1. Regular image encoding >> 2. Grid image encoding >> >> 1. Regular image encoding >> >> Propose to reuse stateful video encoder spec [2]. >> >> - queuing one OUTPUT buffer will produce one CAPTURE buffer. The >> client could trigger Drain sequence at any point of time. >> >> 2. Grid image encoding >> >> Propose to reuse stateful video encoder spec [2]. >> >> - queuing one OUTPUT buffer will produce a number of grids CAPTURE >> buffers. The client could trigger Drain sequence at any point of time. >> >> This image encoding mode is used when the input image resolution is >> bigger then the hardware can support and/or for compatibility reasons >> (for exmaple, the HEIC decoding hardware is not capable to decode higher >> than VGA resolutions). > > Is grid image encoding part of the spec for this format? Is this something > that the venus hardware needs due to image resolution limitations as > described above? Yes, it is part of the ISO/IEC 23008-12 (2017). The spec defines Image grid derivation, where each tile is a separate image and associated with derived image of type _grid_ which reconstruct all tiles into a single image for display. > > Would it be possible for the driver to handle this internally? I.e., > if it detects that it needs to switch to grid mode, can it just encode > each grid and copy it in the capture buffer? This assumes that there is > metadata that can be used by a decoder do find and decode each grid. > In case that is is part of the spec I don't think we have to do it. Something more, when each tile is separate image the decoding process could be done in parallel. >> >> In this mode the input image is divided on square blocks (we call them grids) >> and every block is encoded separately (the Venus driver presently supports >> grid size of 512x512 but that could be changed in the future). >> >> To set the grid size we use a new v4l2 control. > > Can the driver make a choice of the grid size, and the control just > reports the grid size? I.e., does it make sense for userspace to set > this? > I'm not familiar with userspace implementations so far, but my feeling is that the userspace should configure that - at least this will give clients flexibility. References with more information [1] - [5]. > The wiki page [1] doesn't mention grids, so where does this come from? > Is it part of some spec? Or is it a venus-specific feature? > >> >> The side effect of this mode is that the client have to set the v4l2 >> control and thus enable grid encoding before setting the formats on >> CAPTURE and OUTPUT queues, because the grid size reflects on the >> selected resolutions. Also the horizontal and vertical strides will >> also be affected because thеy have to be aligned to the grid size >> in order to satisfy DMA alignment restrictions. >> >> Using of v4l2 control to set up Grid mode and Grid size above looks >> inpractical and somehow breaks the v4l2 and v4l2 control rules, so >> I'd give one more option. >> >> Encoding the Grid mode/size in the new proposed HEIC pixel format: >> >> V4L2_PIX_FMT_HEIC - Regular HEIC image encoding >> V4L2_PIX_FMT_HEIC_GRID_512x512 - Grid HEIC image encoding, grid size of 512x512 >> and so on ... >> >> Comments and suggestions are welcome! > > I notice that this RFC just talks about the encoder, does venus also > support a decoder? How would a HW decoder handle grids? AFAIK the decoding part is not doing something special and reconstructing the whole image from tiles is done by the userspace client [6]. > > Regards, > > Hans -- regards, Stan [1] https://0xc0000054.github.io/pdn-avif/using-image-grids.html#fnref:3 [2] https://nokiatech.github.io/heif/technical.html [3] https://github.com/lclevy/canon_cr3/blob/master/heif.md [4] https://github.com/nokiatech/heif/blob/master/srcs/api-cpp/GridImageItem.cpp [5] https://github.com/strukturag/libheif/blob/master/libheif/heif_context.cc#L163 [6] https://github.com/strukturag/libheif/blob/master/libheif/heif_context.cc#L1317
Le vendredi 11 juin 2021 à 16:12 +0300, Stanimir Varbanov a écrit : > > > > Would it be possible for the driver to handle this internally? I.e., > > if it detects that it needs to switch to grid mode, can it just encode > > each grid and copy it in the capture buffer? This assumes that there is > > metadata that can be used by a decoder do find and decode each grid. > > > > In case that is is part of the spec I don't think we have to do it. > Something more, when each tile is separate image the decoding process > could be done in parallel. Does it means there is no userspace (or at least no Open Source) userspace for it ?
On 6/11/21 5:46 PM, Nicolas Dufresne wrote: > Le vendredi 11 juin 2021 à 16:12 +0300, Stanimir Varbanov a écrit : >>> >>> Would it be possible for the driver to handle this internally? I.e., >>> if it detects that it needs to switch to grid mode, can it just encode >>> each grid and copy it in the capture buffer? This assumes that there is >>> metadata that can be used by a decoder do find and decode each grid. >>> >> >> In case that is is part of the spec I don't think we have to do it. >> Something more, when each tile is separate image the decoding process >> could be done in parallel. > > Does it means there is no userspace (or at least no Open Source) userspace for > it ? > > My searching came to two libraries: * the open source library [1]. * library from Nokia [2]. -- regards, Stan [1] https://github.com/strukturag/libheif [2] https://github.com/nokiatech/heif
Hi, HEIC (High-Efficiency Image Container) is a variant of HEIF (High Efficiency Image File Format) where HEVC/h265 codec is used to encode images. For more info see [1]. In this RFC we propose a new compressed pixel format V4L2_PIX_FMT_HEIC. The name is debatable and could be changed (V4L2_PIX_FMT_HEVC_HEIF is also an option). There are two encoding modes which should be selectable by clients: 1. Regular image encoding 2. Grid image encoding 1. Regular image encoding Propose to reuse stateful video encoder spec [2]. - queuing one OUTPUT buffer will produce one CAPTURE buffer. The client could trigger Drain sequence at any point of time. 2. Grid image encoding Propose to reuse stateful video encoder spec [2]. - queuing one OUTPUT buffer will produce a number of grids CAPTURE buffers. The client could trigger Drain sequence at any point of time. This image encoding mode is used when the input image resolution is bigger then the hardware can support and/or for compatibility reasons (for exmaple, the HEIC decoding hardware is not capable to decode higher than VGA resolutions). In this mode the input image is divided on square blocks (we call them grids) and every block is encoded separately (the Venus driver presently supports grid size of 512x512 but that could be changed in the future). To set the grid size we use a new v4l2 control. The side effect of this mode is that the client have to set the v4l2 control and thus enable grid encoding before setting the formats on CAPTURE and OUTPUT queues, because the grid size reflects on the selected resolutions. Also the horizontal and vertical strides will also be affected because thеy have to be aligned to the grid size in order to satisfy DMA alignment restrictions. Using of v4l2 control to set up Grid mode and Grid size above looks inpractical and somehow breaks the v4l2 and v4l2 control rules, so I'd give one more option. Encoding the Grid mode/size in the new proposed HEIC pixel format: V4L2_PIX_FMT_HEIC - Regular HEIC image encoding V4L2_PIX_FMT_HEIC_GRID_512x512 - Grid HEIC image encoding, grid size of 512x512 and so on ... Comments and suggestions are welcome! regards, Stan [1] https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/dev-encoder.html Stanimir Varbanov (4): media: Add HEIC compressed pixel format v4l2-ctrls: Add HEIC grid size control venus: helpers: Add a new helper for buffer processing venus: Add HEIC image encoder .../media/v4l/pixfmt-compressed.rst | 12 + drivers/media/platform/qcom/venus/Makefile | 2 + drivers/media/platform/qcom/venus/core.h | 10 + drivers/media/platform/qcom/venus/helpers.c | 20 + drivers/media/platform/qcom/venus/helpers.h | 1 + drivers/media/platform/qcom/venus/hfi_cmds.c | 10 +- .../media/platform/qcom/venus/hfi_helper.h | 5 + drivers/media/platform/qcom/venus/ienc.c | 1348 +++++++++++++++++ drivers/media/platform/qcom/venus/ienc.h | 14 + .../media/platform/qcom/venus/ienc_ctrls.c | 83 + drivers/media/v4l2-core/v4l2-ctrls.c | 3 + drivers/media/v4l2-core/v4l2-ioctl.c | 1 + include/uapi/linux/v4l2-controls.h | 1 + include/uapi/linux/videodev2.h | 1 + 14 files changed, 1510 insertions(+), 1 deletion(-) create mode 100644 drivers/media/platform/qcom/venus/ienc.c create mode 100644 drivers/media/platform/qcom/venus/ienc.h create mode 100644 drivers/media/platform/qcom/venus/ienc_ctrls.c -- 2.25.1