avformat/matroskadec: Fix lzo decompression

When a Matroska Block is only stored in compressed form, the size of
the uncompressed block is not explicitly coded and therefore not known
before decompressing it. Therefore the demuxer uses a guess for the
uncompressed size: The first guess is three times the compressed size
and if this is not enough, it is repeatedly incremented by a factor of
three. But when this happens with lzo, the decompression is neither
resumed nor started again. Instead when av_lzo1x_decode indicates that x
bytes of input data could not be decoded, because the output buffer is
already full, the first (not the last) x bytes of the input buffer are
resent for decoding in the next try; they overwrite already decoded
data.

This commit fixes this by instead restarting the decompression anew,
just with a bigger buffer.

This seems to be a regression since 935ec5a1.

A FATE-test for this has been added.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: James Almer <jamrial@gmail.com>

1 files changed, 2 insertions, 1 deletions

diff --git a/libavformat/matroskadec.c b/libavformat/matroskadec.c
index 72624dc3f1..75f72d330c 100644
--- a/libavformat/matroskadec.c
+++ b/libavformat/matroskadec.c
@@ -1599,6 +1599,7 @@ static int matroska_decode_buffer(uint8_t **buf, int *buf_size,
 #if CONFIG_LZO
     case MATROSKA_TRACK_ENCODING_COMP_LZO:
         do {
+            int insize = isize;
             olen       = pkt_size *= 3;
             newpktdata = av_realloc(pkt_data, pkt_size + AV_LZO_OUTPUT_PADDING
                                                        + AV_INPUT_BUFFER_PADDING_SIZE);
@@ -1607,7 +1608,7 @@ static int matroska_decode_buffer(uint8_t **buf, int *buf_size,
                 goto failed;
             }
             pkt_data = newpktdata;
-            result   = av_lzo1x_decode(pkt_data, &olen, data, &isize);
+            result   = av_lzo1x_decode(pkt_data, &olen, data, &insize);
         } while (result == AV_LZO_OUTPUT_FULL && pkt_size < 10000000);
         if (result) {
             result = AVERROR_INVALIDDATA;