lavu: fix memory leaks by using a mutex instead of atomics

The buffer pool has to atomically add and remove entries from the linked
list of available buffers. This was done by removing the entire list
with a CAS operation, working on it, and then setting it back again
(using a retry-loop in case another thread was doing the same thing).

This could effectively cause memory leaks: while a thread was working on
the buffer list, other threads would allocate new buffers, increasing
the pool's total size. There was no real leak, but since these extra
buffers were not needed, but not free'd either (except when the buffer
pool was destroyed), this had the same effects as a real leak. For some
reason, growth was exponential, and could easily kill the process due
to OOM in real-world uses.

Fix this by using a mutex to protect the list operations. The fancy
way atomics remove the whole list to work on it is not needed anymore,
which also avoids the situation which was causing the leak.

Signed-off-by: Anton Khirnov <anton@khirnov.net>

1 files changed, 4 insertions, 2 deletions

diff --git a/libavutil/buffer_internal.h b/libavutil/buffer_internal.h
index cce83c3cd1..1032a543e5 100644
--- a/libavutil/buffer_internal.h
+++ b/libavutil/buffer_internal.h
@@ -22,6 +22,7 @@
 #include <stdint.h>
 
 #include "buffer.h"
+#include "thread.h"
 
 /**
  * The buffer is always treated as read-only.
@@ -68,11 +69,12 @@ typedef struct BufferPoolEntry {
     void (*free)(void *opaque, uint8_t *data);
 
     AVBufferPool *pool;
-    struct BufferPoolEntry * volatile next;
+    struct BufferPoolEntry *next;
 } BufferPoolEntry;
 
 struct AVBufferPool {
-    BufferPoolEntry * volatile pool;
+    AVMutex mutex;
+    BufferPoolEntry *pool;
 
     /*
      * This is used to track when the pool is to be freed.