avformat: Redo cleanup of demuxers upon read_header() failure

If reading the header fails, the demuxer's read_close() function (if
existing) is not called automatically; instead several demuxers call it
via "goto fail" in read_header().

This commit intends to change this by adding an internal flag for
demuxers that can be used to set on a per-AVInputFormat basis whether
read_close() should be called generically after an error during
read_header().

The flag controlling this behaviour needs to be added because it might
be unsafe to call read_close() generally (e.g. this might lead to
read_close() being called twice and this might e.g. lead to double-frees
if av_free() is used instead of av_freep(); or a size field has not
been reset after freeing the elements (see the mov demuxer for an
example of this)).

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>

1 files changed, 4 insertions, 1 deletions

diff --git a/libavformat/utils.c b/libavformat/utils.c
index 0df14682a4..998fddf270 100644
--- a/libavformat/utils.c
+++ b/libavformat/utils.c
@@ -566,8 +566,11 @@ int avformat_open_input(AVFormatContext **ps, const char *filename,
         ff_id3v2_read_dict(s->pb, &s->internal->id3v2_meta, ID3v2_DEFAULT_MAGIC, &id3v2_extra_meta);
 
     if (s->iformat->read_header)
-        if ((ret = s->iformat->read_header(s)) < 0)
+        if ((ret = s->iformat->read_header(s)) < 0) {
+            if (s->iformat->flags_internal & FF_FMT_INIT_CLEANUP)
+                goto close;
             goto fail;
+        }
 
     if (!s->metadata) {
         s->metadata = s->internal->id3v2_meta;