avformat: Redo cleanup of demuxers upon read_header() failure

If reading the header fails, the demuxer's read_close() function (if
existing) is not called automatically; instead several demuxers call it
via "goto fail" in read_header().

This commit intends to change this by adding an internal flag for
demuxers that can be used to set on a per-AVInputFormat basis whether
read_close() should be called generically after an error during
read_header().

The flag controlling this behaviour needs to be added because it might
be unsafe to call read_close() generally (e.g. this might lead to
read_close() being called twice and this might e.g. lead to double-frees
if av_free() is used instead of av_freep(); or a size field has not
been reset after freeing the elements (see the mov demuxer for an
example of this)).

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>

1 files changed, 6 insertions, 0 deletions

diff --git a/libavformat/internal.h b/libavformat/internal.h
index c6adf7b872..240de9e289 100644
--- a/libavformat/internal.h
+++ b/libavformat/internal.h
@@ -42,6 +42,12 @@
 #    define hex_dump_debug(class, buf, size) do { if (0) av_hex_dump_log(class, AV_LOG_DEBUG, buf, size); } while(0)
 #endif
 
+/**
+ * For an AVInputFormat with this flag set read_close() needs to be called
+ * by the caller upon read_header() failure.
+ */
+#define FF_FMT_INIT_CLEANUP                             (1 << 0)
+
 typedef struct AVCodecTag {
     enum AVCodecID id;
     unsigned int tag;