avformat/rm: limit packet size

The chunk size is limited to 0xFFFF (written by avio_wb16), so make
sure that the packet size is not too large.

Such large frames need to be split into slices smaller than 64 kB, but
that is currently supported neither by the rv10/rv20 encoders nor the rm
muxer.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

See Ticket244

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

1 files changed, 5 insertions, 0 deletions

diff --git a/libavformat/rmenc.c b/libavformat/rmenc.c
index 838388f30d..8763fff235 100644
--- a/libavformat/rmenc.c
+++ b/libavformat/rmenc.c
@@ -396,6 +396,11 @@ static int rm_write_video(AVFormatContext *s, const uint8_t *buf, int size, int
     /* Well, I spent some time finding the meaning of these bits. I am
        not sure I understood everything, but it works !! */
 #if 1
+    /* 0xFFFF is the maximal chunk size; header needs at most 7 + 4 + 12 B */
+    if (size > 0xFFFF - 7 - 4 - 12) {
+        av_log(s, AV_LOG_ERROR, "large packet size %d not supported\n", size);
+        return AVERROR_PATCHWELCOME;
+    }
     write_packet_header(s, stream, size + 7 + (size >= 0x4000)*4, key_frame);
     /* bit 7: '1' if final packet of a frame converted in several packets */
     avio_w8(pb, 0x81);