h264_mp4toannexb: Try to avoid four byte startcodes

According to the H.264 specifications, the only NAL units that need to
have four byte startcodes in H.264 Annex B format are SPS/PPS units and
units that start a new access unit. Before af7e953a, the first of these
conditions wasn't upheld as already existing in-band parameter sets
would not automatically be written with a four byte startcode, but only
when they already were at the beginning of their input packets. But it
made four byte startcodes be used too often as every unit that is written
together with a parameter set that is inserted from extradata received a
four byte startcode although a three byte start code would suffice
unless the unit itself were a parameter set.

FATE has been updated to reflect the changes. Although the patch leaves
the extradata unchanged, the size of the extradata according to the FATE
reports changes. This is due to a quirk in ff_h2645_packet_split which
is used by extract_extradata: If the input is Annex B, the first zero of
a four byte startcode is considered a part of the last unit (if any).

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

1 files changed, 3 insertions, 3 deletions

diff --git a/libavcodec/h264_mp4toannexb_bsf.c b/libavcodec/h264_mp4toannexb_bsf.c
index 561126c997..9aed3f1435 100644
--- a/libavcodec/h264_mp4toannexb_bsf.c
+++ b/libavcodec/h264_mp4toannexb_bsf.c
@@ -47,7 +47,7 @@ static int alloc_and_copy(AVPacket *out,
                           const uint8_t *in, uint32_t in_size, int ps)
 {
     uint32_t offset         = out->size;
-    uint8_t start_code_size = offset == 0 || ps ? 4 : 3;
+    uint8_t start_code_size = offset == 0 && sps_pps_size == 0 || ps ? 4 : 3;
     int err;
 
     err = av_grow_packet(out, sps_pps_size + in_size + start_code_size);
@@ -244,7 +244,7 @@ static int h264_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out)
         if (s->new_idr && unit_type == H264_NAL_IDR_SLICE && !s->idr_sps_seen && !s->idr_pps_seen) {
             if ((ret=alloc_and_copy(out,
                                ctx->par_out->extradata, ctx->par_out->extradata_size,
-                               buf, nal_size, 1)) < 0)
+                               buf, nal_size, 0)) < 0)
                 goto fail;
             s->new_idr = 0;
         /* if only SPS has been seen, also insert PPS */
@@ -255,7 +255,7 @@ static int h264_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out)
                     goto fail;
             } else if ((ret = alloc_and_copy(out,
                                         s->pps, s->pps_size,
-                                        buf, nal_size, 1)) < 0)
+                                        buf, nal_size, 0)) < 0)
                 goto fail;
         } else {
             if ((ret=alloc_and_copy(out, NULL, 0, buf, nal_size, unit_type == H264_NAL_SPS || unit_type == H264_NAL_PPS)) < 0)