diff mbox

[v4,00/24] H.264 Field Decoding Support for Frame-based Decoders

Message ID 20220426125751.108293-1-nicolas.dufresne@collabora.com
State Superseded
Headers show

Commit Message

Nicolas Dufresne April 26, 2022, 12:57 p.m. UTC
Until now, only Cedrus (a slice base decoder) supported interlaced
decoding.  In order to support field decoding in our frame-based decoder,
the v4l2-h264 library needed adaptation to produce the appropriate
reference lists.

This patch extends the v4l2-h264 library to produce the larger references
list needed to represent fields separately. Hantro, MTK-VCODEC and RKVDEC
drivers have been adapted to accommodate the larger lists. Though, only
Hantro and RKVDEC actually have HW support for field decoding. So only
these two have been updated to make use of the larger lists. All this work
has been done using the H.264 specification, LibreELEC downstream kernel
patches, Rockchip MPP reference software and Hantro reference software.

For reviewers, the following is the map of all commit. Patches that could
be merge independently of this serie are marked as independent. Note that
the test results do depend on the generic fixes.

01    : Documentation fix (independent)
02-03 : Improving some generic traces (independent)
04    : Minor v4l2-h264 fix (independent)
05-11 : v4l2-h264 field decoding support
12-16 : rkvdec h.264 generic fixes (independent)
17-20 : rkvdec h.264 field decoding support
21-24 : hantro h.264 field decoding support

All this work have been tested using GStreamer mainline implementation
but also with FFMPEG LibreELEC fork using the testing tool fluster
running through the ITU-T H.264 (2016-02) AVCv2 set of bitstream. Before
this patch, the scores were:

Hantro:
  FFMPEG:   88/135
  GSteamer: 90/135
RKVDEC:
  FFMPEG:   73/135
  GSteamer: 77/135

And after these changes:

Hantro:
  FFMPEG:   118/135
  GSteamer: 129/135
RKVDEC:
  FFMPEG:   118/135
  GSteamer: 129/135

Note that a bug in FFMPEG / LibreELEC fork was noticed and fixed with the
following change:


Some useful links:

Detailed Hantro Results:     https://gitlab.freedesktop.org/-/snippets/5189
Detailed RKVDEC Results:     https://gitlab.freedesktop.org/-/snippets/5253
ITU-T H.264 (2016-02) AVCv2: https://www.itu.int/net/itu-t/sigdb/spevideo/VideoForm-s.aspx?val=102002641
Fluster:                     https://github.com/fluendo/fluster
GStreamer:                   https://gitlab.freedesktop.org/gstreamer/gstreamer/
FFMPEG Fork:                 https://github.com/jernejsk/FFmpeg/tree/v4l2-request-hwaccel-4.4
Rockchip MPP:                https://github.com/rockchip-linux/mpp

Changes in v4:
- Fixed mtk-vcodec port
- Ported tegra-vde driver (compiled tested only)
- Applied Hans' commit message, comment and documentation suggestions

Changes in v3:
- Improved debug message on timestamp miss-match
- Moved H264 SPS validation into rkvdec-h264
- Added more comments around H264 SPS validation
- Also validate at streamon (rkvdec start())
- Applied more Review-by and Fixes tag
- Fixed Signed-off-by chain in Jonas patch

Changes in v2:
- Applied most of Sebastian's suggestion in comments and commit messages.
- Use a bool for dpb_valid and dpb_bottom in rkvdec
- Dropped one wrong typo fix (media: v4l2-mem2mem: Fix typo in trace message)
- Dropped Alex fix (media: rkvdec-h264: Don't hardcode SPS/PPS parameters
  + I will carry this one later, it seems cosmetic

Jonas Karlman (5):
  media: rkvdec: h264: Fix bit depth wrap in pps packet
  media: rkvdec: h264: Validate and use pic width and height in mbs
  media: rkvdec: h264: Fix reference frame_num wrap for second field
  media: rkvdec: Ensure decoded resolution fit coded resolution
  media: hantro: h264: Make dpb entry management more robust

Nicolas Dufresne (18):
  media: doc: Document dual use of H.264 pic_num/frame_num
  media: v4l2-mem2mem: Trace on implicit un-hold
  media: h264: Avoid wrapping long_term_frame_idx
  media: h264: Use v4l2_h264_reference for reflist
  media: h264: Increase reference lists size to 32
  media: h264: Store current picture fields
  media: h264: Store all fields into the unordered list
  media: v4l2: Trace calculated p/b0/b1 initial reflist
  media: h264: Sort p/b reflist using frame_num
  media: v4l2: Reorder field reflist
  media: rkvdec: Stop overclocking the decoder
  media: rkvdec: h264: Fix dpb_valid implementation
  media: rkvdec: Move H264 SPS validation in rkvdec-h264
  media: rkvdec-h264: Add field decoding support
  media: rkvdec: Enable capture buffer holding for H264
  media: hantro: Stop using H.264 parameter pic_num
  media: hantro: Add H.264 field decoding support
  media: hantro: Enable HOLD_CAPTURE_BUF for H.264

Sebastian Fricke (1):
  media: videobuf2-v4l2: Warn on holding buffers without support

 .../media/v4l/ext-ctrls-codec-stateless.rst   |  10 +-
 .../media/common/videobuf2/videobuf2-v4l2.c   |   7 +-
 .../mediatek/vcodec/vdec/vdec_h264_req_if.c   |  36 ++-
 .../media/platform/nvidia/tegra-vde/h264.c    |  19 +-
 drivers/media/v4l2-core/v4l2-h264.c           | 275 +++++++++++++++---
 drivers/media/v4l2-core/v4l2-mem2mem.c        |   1 +
 .../staging/media/hantro/hantro_g1_h264_dec.c |  38 +--
 drivers/staging/media/hantro/hantro_h264.c    | 134 +++++++--
 drivers/staging/media/hantro/hantro_hw.h      |   7 +-
 drivers/staging/media/hantro/hantro_v4l2.c    |  25 ++
 .../media/hantro/rockchip_vpu2_hw_h264_dec.c  |  98 +++----
 drivers/staging/media/rkvdec/rkvdec-h264.c    | 157 +++++++---
 drivers/staging/media/rkvdec/rkvdec.c         |  35 +--
 drivers/staging/media/rkvdec/rkvdec.h         |   2 +
 include/media/v4l2-h264.h                     |  31 +-
 15 files changed, 639 insertions(+), 236 deletions(-)

Comments

Hans Verkuil April 28, 2022, 6:12 a.m. UTC | #1
On 27/04/2022 17:08, Nicolas Dufresne wrote:
> Le mercredi 27 avril 2022 à 13:31 +0900, Tomasz Figa a écrit :
>> Hi Nicolas, Sebastian,
>>
>> On Tue, Apr 26, 2022 at 9:58 PM Nicolas Dufresne
>> <nicolas.dufresne@collabora.com> wrote:
>>>
>>> From: Sebastian Fricke <sebastian.fricke@collabora.com>
>>>
>>> Using V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF flag without specifying the
>>> subsystem flag VB2_V4L2_FL_SUPPORTS_M2M_HOLD_CAPTURE_BUF, results in
>>> silently ignoring it.
>>> Warn the user via a debug print when the flag is requested but ignored
>>> by the videobuf2 framework.
>>>
>>> Signed-off-by: Sebastian Fricke <sebastian.fricke@collabora.com>
>>> Signed-off-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
>>> Reviewed-by: Ezequiel Garcia <ezequiel@vanguardiasur.com.ar>
>>> ---
>>>  drivers/media/common/videobuf2/videobuf2-v4l2.c | 7 ++++++-
>>>  1 file changed, 6 insertions(+), 1 deletion(-)
>>>
>>
>> Thanks for the patch. Please see my comments inline.
>>
>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>>> index 6edf4508c636..812c8d1962e0 100644
>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>>> @@ -329,8 +329,13 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
>>>                  */
>>>                 vbuf->flags &= ~V4L2_BUF_FLAG_TIMECODE;
>>>                 vbuf->field = b->field;
>>> -               if (!(q->subsystem_flags & VB2_V4L2_FL_SUPPORTS_M2M_HOLD_CAPTURE_BUF))
>>> +               if (!(q->subsystem_flags & VB2_V4L2_FL_SUPPORTS_M2M_HOLD_CAPTURE_BUF)) {
>>> +                       if (vbuf->flags & V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF)
>>> +                               dprintk(q, 1,
>>> +                                       "Request holding buffer (%d), unsupported on output queue\n",
>>> +                                       b->index);
>>
>> I wonder if we shouldn't just fail such a QBUF operation. Otherwise
>> the application would get unexpected behavior from the kernel.
>> Although it might be too late to do it now if there are applications
>> that rely on this implicit ignore...
> 
> In the context of this patchset, the statu quo seems to be the logical thing to
> do. We can raise this up in a separate thread. The side effect is of course
> confusing for developers, but it is hard for me to tell if a hard failure may
> break an existing software.

I am leaning towards returning an error as well. It makes no sense to try
to hold on to a buffer when this is not supported.

I also thought that it should be enough to rely on the core to clear the
flag upon return if it isn't supported, but looking through the vb2 core code
it looks like we're not clearing unknown flags at all, so running this for
older kernels that do not support holding at all will not clear the flag
either.

The handling for flags in vb2 can be improved, I think I'll take a look at
that myself.

I plan to merge this series soon, but will skip this patch for now.

Regards,

	Hans

> 
> regards,
> Nicolas
> 
>>
>> Best regards,
>> Tomasz
>
Nicolas Dufresne April 28, 2022, 1:09 p.m. UTC | #2
Le jeudi 28 avril 2022 à 08:12 +0200, Hans Verkuil a écrit :
> On 27/04/2022 17:08, Nicolas Dufresne wrote:
> > Le mercredi 27 avril 2022 à 13:31 +0900, Tomasz Figa a écrit :
> > > Hi Nicolas, Sebastian,
> > > 
> > > On Tue, Apr 26, 2022 at 9:58 PM Nicolas Dufresne
> > > <nicolas.dufresne@collabora.com> wrote:
> > > > 
> > > > From: Sebastian Fricke <sebastian.fricke@collabora.com>
> > > > 
> > > > Using V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF flag without specifying the
> > > > subsystem flag VB2_V4L2_FL_SUPPORTS_M2M_HOLD_CAPTURE_BUF, results in
> > > > silently ignoring it.
> > > > Warn the user via a debug print when the flag is requested but ignored
> > > > by the videobuf2 framework.
> > > > 
> > > > Signed-off-by: Sebastian Fricke <sebastian.fricke@collabora.com>
> > > > Signed-off-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
> > > > Reviewed-by: Ezequiel Garcia <ezequiel@vanguardiasur.com.ar>
> > > > ---
> > > >  drivers/media/common/videobuf2/videobuf2-v4l2.c | 7 ++++++-
> > > >  1 file changed, 6 insertions(+), 1 deletion(-)
> > > > 
> > > 
> > > Thanks for the patch. Please see my comments inline.
> > > 
> > > > diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > > > index 6edf4508c636..812c8d1962e0 100644
> > > > --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > > > +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > > > @@ -329,8 +329,13 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
> > > >                  */
> > > >                 vbuf->flags &= ~V4L2_BUF_FLAG_TIMECODE;
> > > >                 vbuf->field = b->field;
> > > > -               if (!(q->subsystem_flags & VB2_V4L2_FL_SUPPORTS_M2M_HOLD_CAPTURE_BUF))
> > > > +               if (!(q->subsystem_flags & VB2_V4L2_FL_SUPPORTS_M2M_HOLD_CAPTURE_BUF)) {
> > > > +                       if (vbuf->flags & V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF)
> > > > +                               dprintk(q, 1,
> > > > +                                       "Request holding buffer (%d), unsupported on output queue\n",
> > > > +                                       b->index);
> > > 
> > > I wonder if we shouldn't just fail such a QBUF operation. Otherwise
> > > the application would get unexpected behavior from the kernel.
> > > Although it might be too late to do it now if there are applications
> > > that rely on this implicit ignore...
> > 
> > In the context of this patchset, the statu quo seems to be the logical thing to
> > do. We can raise this up in a separate thread. The side effect is of course
> > confusing for developers, but it is hard for me to tell if a hard failure may
> > break an existing software.
> 
> I am leaning towards returning an error as well. It makes no sense to try
> to hold on to a buffer when this is not supported.
> 
> I also thought that it should be enough to rely on the core to clear the
> flag upon return if it isn't supported, but looking through the vb2 core code
> it looks like we're not clearing unknown flags at all, so running this for
> older kernels that do not support holding at all will not clear the flag
> either.
> 
> The handling for flags in vb2 can be improved, I think I'll take a look at
> that myself.
> 
> I plan to merge this series soon, but will skip this patch for now.

Ok, no problem. For me, as long as we do something about it, since it was not
obvious and time consuming to debug.

regards,
Nicolas

> 
> Regards,
> 
> 	Hans
> 
> > 
> > regards,
> > Nicolas
> > 
> > > 
> > > Best regards,
> > > Tomasz
> > 
>
diff mbox

Patch

diff --git a/libavcodec/v4l2_request_h264.c b/libavcodec/v4l2_request_h264.c
index 88da8f0a2d..394bae0550 100644
--- a/libavcodec/v4l2_request_h264.c
+++ b/libavcodec/v4l2_request_h264.c
@@ -66,7 +66,7 @@  static void fill_dpb_entry(struct v4l2_h264_dpb_entry *entry, const H264Picture
 {
     entry->reference_ts = ff_v4l2_request_get_capture_timestamp(pic->f);
     entry->pic_num = pic->pic_id;
-    entry->frame_num = pic->frame_num;
+    entry->frame_num = pic->long_ref ? pic->pic_id : pic->frame_num;
     entry->fields = pic->reference & V4L2_H264_FRAME_REF;
     entry->flags = V4L2_H264_DPB_ENTRY_FLAG_VALID;
     if (entry->fields)