avformat/matroskadec: Fix buffer overflow when demuxing RealAudio 28.8

RealAudio 28.8 (like other RealAudio codecs) uses a special demuxing
mode in which the data of the existing Matroska Blocks is not simply
forwarded as-is. Instead data from several Blocks is recombined
together to output several packets. The parameters governing this
process are parsed from the CodecPrivate: Coded framesize (cfs), frame
size (w) and sub_packet_h (h).

During demuxing, h/2 pieces of data of size cfs each are read from every
Matroska (Simple)Block and put at offset m * 2 * w + n * cfs of a buffer
of size h * w, where m ranges from 0 to h/2 - 1 for each Block while n
is initially zero and incremented after a Block has been parsed until it
is h, at which poin the assembled packets are output and n reset.

The highest offset is given by (h/2 - 1) * 2 * w + (h - 1) * cfs + cfs
while the destination buffer's size is given by h * w. For even h, this
leads to a buffer overflow (and potential segfault) if h * cfs > 2 * w;
for odd h, the condition is h * cfs > 3 * w.

This commit adds a check to rule this out.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

1 files changed, 3 insertions, 0 deletions

diff --git a/libavformat/matroskadec.c b/libavformat/matroskadec.c
index 844f96cd52..951695b5b5 100644
--- a/libavformat/matroskadec.c
+++ b/libavformat/matroskadec.c
@@ -2612,6 +2612,9 @@ static int matroska_parse_tracks(AVFormatContext *s)
                 return AVERROR_INVALIDDATA;
 
             if (codec_id == AV_CODEC_ID_RA_288) {
+                if ((int64_t)track->audio.sub_packet_h * track->audio.coded_framesize
+                    > (2 + (track->audio.sub_packet_h & 1)) * track->audio.frame_size)
+                    return AVERROR_INVALIDDATA;
                 st->codecpar->block_align = track->audio.coded_framesize;
                 track->codec_priv.size = 0;
             } else {