avcodec/hevc_mp4toannexb_bsf: Check NAL size against available input

The hevc_mp4toannexb bsf does not explicitly check whether a NAL unit
is so big that it extends beyond the end of the input packet; it does so
only implicitly by using the checked version of the bytestream2 API.
But this has downsides compared to real checks: It can lead to huge
allocations (up to 2GiB) even when the input packet is just a few bytes.
And furthermore it leads to uninitialized data being output.
So add a check to error out early if it happens.

Also check directly whether there is enough data for the length field.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

1 files changed, 5 insertions, 1 deletions

diff --git a/libavcodec/hevc_mp4toannexb_bsf.c b/libavcodec/hevc_mp4toannexb_bsf.c
index a880d9ba9a..ba1deb2848 100644
--- a/libavcodec/hevc_mp4toannexb_bsf.c
+++ b/libavcodec/hevc_mp4toannexb_bsf.c
@@ -142,10 +142,14 @@ static int hevc_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out)
         int      nalu_type;
         int is_irap, add_extradata, extra_size, prev_size;
 
+        if (bytestream2_get_bytes_left(&gb) < s->length_size) {
+            ret = AVERROR_INVALIDDATA;
+            goto fail;
+        }
         for (i = 0; i < s->length_size; i++)
             nalu_size = (nalu_size << 8) | bytestream2_get_byte(&gb);
 
-        if (nalu_size < 2) {
+        if (nalu_size < 2 || nalu_size > bytestream2_get_bytes_left(&gb)) {
             ret = AVERROR_INVALIDDATA;
             goto fail;
         }