avformat: Redo cleanup of demuxers upon read_header() failure

If reading the header fails, the demuxer's read_close() function (if
existing) is not called automatically; instead several demuxers call it
via "goto fail" in read_header().

This commit intends to change this by adding an internal flag for
demuxers that can be used to set on a per-AVInputFormat basis whether
read_close() should be called generically after an error during
read_header().

The flag controlling this behaviour needs to be added because it might
be unsafe to call read_close() generally (e.g. this might lead to
read_close() being called twice and this might e.g. lead to double-frees
if av_free() is used instead of av_freep(); or a size field has not
been reset after freeing the elements (see the mov demuxer for an
example of this)).

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>

2 files changed, 10 insertions, 1 deletions

diff --git a/libavformat/internal.h b/libavformat/internal.h
index c6adf7b872..240de9e289 100644
--- a/libavformat/internal.h
+++ b/libavformat/internal.h
@@ -42,6 +42,12 @@
 #    define hex_dump_debug(class, buf, size) do { if (0) av_hex_dump_log(class, AV_LOG_DEBUG, buf, size); } while(0)
 #endif
 
+/**
+ * For an AVInputFormat with this flag set read_close() needs to be called
+ * by the caller upon read_header() failure.
+ */
+#define FF_FMT_INIT_CLEANUP                             (1 << 0)
+
 typedef struct AVCodecTag {
     enum AVCodecID id;
     unsigned int tag;
diff --git a/libavformat/utils.c b/libavformat/utils.c
index 0df14682a4..998fddf270 100644
--- a/libavformat/utils.c
+++ b/libavformat/utils.c
@@ -566,8 +566,11 @@ int avformat_open_input(AVFormatContext **ps, const char *filename,
         ff_id3v2_read_dict(s->pb, &s->internal->id3v2_meta, ID3v2_DEFAULT_MAGIC, &id3v2_extra_meta);
 
     if (s->iformat->read_header)
-        if ((ret = s->iformat->read_header(s)) < 0)
+        if ((ret = s->iformat->read_header(s)) < 0) {
+            if (s->iformat->flags_internal & FF_FMT_INIT_CLEANUP)
+                goto close;
             goto fail;
+        }
 
     if (!s->metadata) {
         s->metadata = s->internal->id3v2_meta;