avutil/mem: Handle fast allocations near UINT_MAX properly

av_fast_realloc and av_fast_mallocz? store the size of
the objects they allocate in an unsigned. Yet they overallocate
and currently they can allocate more than UINT_MAX bytes
in case a user has requested a size of about UINT_MAX * 16 / 17
or more if SIZE_MAX > UINT_MAX (and if the user increased
max_alloc_size via av_max_alloc). In this case it is impossible
to store the true size of the buffer via the unsigned*;
future requests are likely to use the (re)allocation codepath
even if the buffer is actually large enough because of
the incorrect size.

Fix this by ensuring that the actually allocated size
always fits into an unsigned. (This entails erroring out
in case the user requested more than UINT_MAX.)

Reviewed-by: Tomas Härdin <tjoppen@acc.umu.se>
Reviewed-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>

1 files changed, 4 insertions, 0 deletions

diff --git a/libavutil/mem.c b/libavutil/mem.c
index a0c9a42849..18aff5291f 100644
--- a/libavutil/mem.c
+++ b/libavutil/mem.c
@@ -510,6 +510,8 @@ void *av_fast_realloc(void *ptr, unsigned int *size, size_t min_size)
         return ptr;
 
     max_size = atomic_load_explicit(&max_alloc_size, memory_order_relaxed);
+    /* *size is an unsigned, so the real maximum is <= UINT_MAX. */
+    max_size = FFMIN(max_size, UINT_MAX);
 
     if (min_size > max_size) {
         *size = 0;
@@ -542,6 +544,8 @@ static inline void fast_malloc(void *ptr, unsigned int *size, size_t min_size, i
     }
 
     max_size = atomic_load_explicit(&max_alloc_size, memory_order_relaxed);
+    /* *size is an unsigned, so the real maximum is <= UINT_MAX. */
+    max_size = FFMIN(max_size, UINT_MAX);
 
     if (min_size > max_size) {
         av_freep(ptr);