avformat/matroskaenc: Make ebml_num_size() more robust

Matroska (or actually EBML) uses variable-length numbers where only
seven bits of every byte is usable for the length; the other bits encode
the length of the variable-length number. So in order to find out how
many bytes one needs to encode a given number one can use a loop like
while (num >> 7 * bytes) bytes++; the Matroska muxer effectively did this.

Yet it has a disadvantage: It is impossible for the result of a single
right shift of an unsigned number with most significant bit set to be
zero, because one can only shift by 0..(width - 1). On some
architectures like x64 it is not even possible to do it with undefined
right shifts in which case this leads to an infinite loop.

This can be easily avoided by switching to a loop whose condition is
(num >>= 7). The maximum value the so modified function can return
is 10; any value > 8 is invalid and will now lead to an assert in
put_ebml_num() or in start_ebml_master() (or actually in
put_ebml_size_unknown()).

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

1 files changed, 4 insertions, 2 deletions

diff --git a/libavformat/matroskaenc.c b/libavformat/matroskaenc.c
index e1ddf366d4..1a3db72043 100644
--- a/libavformat/matroskaenc.c
+++ b/libavformat/matroskaenc.c
@@ -194,9 +194,11 @@ static void put_ebml_size_unknown(AVIOContext *pb, int bytes)
  */
 static int ebml_num_size(uint64_t num)
 {
-    int bytes = 1;
-    while ((num + 1) >> bytes * 7)
+    int bytes = 0;
+    num++;
+    do {
         bytes++;
+    } while (num >>= 7);
     return bytes;
 }