diff options
Diffstat (limited to 'doc/filter_design.txt')
| -rw-r--r-- | doc/filter_design.txt | 288 |
1 files changed, 288 insertions, 0 deletions
diff --git a/doc/filter_design.txt b/doc/filter_design.txt new file mode 100644 index 0000000000..885b19b6f6 --- /dev/null +++ b/doc/filter_design.txt @@ -0,0 +1,288 @@ +Filter design +============= + +This document explains guidelines that should be observed (or ignored with +good reason) when writing filters for libavfilter. + +In this document, the word “frame” indicates either a video frame or a group +of audio samples, as stored in an AVFrame structure. + + +Format negotiation +================== + + The query_formats method should set, for each input and each output links, + the list of supported formats. + + For video links, that means pixel format. For audio links, that means + channel layout, sample format (the sample packing is implied by the sample + format) and sample rate. + + The lists are not just lists, they are references to shared objects. When + the negotiation mechanism computes the intersection of the formats + supported at each end of a link, all references to both lists are replaced + with a reference to the intersection. And when a single format is + eventually chosen for a link amongst the remaining list, again, all + references to the list are updated. + + That means that if a filter requires that its input and output have the + same format amongst a supported list, all it has to do is use a reference + to the same list of formats. + + query_formats can leave some formats unset and return AVERROR(EAGAIN) to + cause the negotiation mechanism to try again later. That can be used by + filters with complex requirements to use the format negotiated on one link + to set the formats supported on another. + + +Frame references ownership and permissions +========================================== + + Principle + --------- + + Audio and video data are voluminous; the frame and frame reference + mechanism is intended to avoid, as much as possible, expensive copies of + that data while still allowing the filters to produce correct results. + + The data is stored in buffers represented by AVFrame structures. + Several references can point to the same frame buffer; the buffer is + automatically deallocated once all corresponding references have been + destroyed. + + The characteristics of the data (resolution, sample rate, etc.) are + stored in the reference; different references for the same buffer can + show different characteristics. In particular, a video reference can + point to only a part of a video buffer. + + A reference is usually obtained as input to the filter_frame method or + requested using the ff_get_video_buffer or ff_get_audio_buffer + functions. A new reference on an existing buffer can be created with + av_frame_ref(). A reference is destroyed using + the av_frame_free() function. + + Reference ownership + ------------------- + + At any time, a reference “belongs” to a particular piece of code, + usually a filter. With a few caveats that will be explained below, only + that piece of code is allowed to access it. It is also responsible for + destroying it, although this is sometimes done automatically (see the + section on link reference fields). + + Here are the (fairly obvious) rules for reference ownership: + + * A reference received by the filter_frame method belongs to the + corresponding filter. + + * A reference passed to ff_filter_frame is given away and must no longer + be used. + + * A reference created with av_frame_ref() belongs to the code that + created it. + + * A reference obtained with ff_get_video_buffer or ff_get_audio_buffer + belongs to the code that requested it. + + * A reference given as return value by the get_video_buffer or + get_audio_buffer method is given away and must no longer be used. + + Link reference fields + --------------------- + + The AVFilterLink structure has a few AVFrame fields. + + partial_buf is used by libavfilter internally and must not be accessed + by filters. + + fifo contains frames queued in the filter's input. They belong to the + framework until they are taken by the filter. + + Reference permissions + --------------------- + + Since the same frame data can be shared by several frames, modifying may + have unintended consequences. A frame is considered writable if only one + reference to it exists. The code owning that reference it then allowed + to modify the data. + + A filter can check if a frame is writable by using the + av_frame_is_writable() function. + + A filter can ensure that a frame is writable at some point of the code + by using the ff_inlink_make_frame_writable() function. It will duplicate + the frame if needed. + + A filter can ensure that the frame passed to the filter_frame() callback + is writable by setting the needs_writable flag on the corresponding + input pad. It does not apply to the activate() callback. + + +Frame scheduling +================ + + The purpose of these rules is to ensure that frames flow in the filter + graph without getting stuck and accumulating somewhere. + + Simple filters that output one frame for each input frame should not have + to worry about it. + + There are two design for filters: one using the filter_frame() and + request_frame() callbacks and the other using the activate() callback. + + The design using filter_frame() and request_frame() is legacy, but it is + suitable for filters that have a single input and process one frame at a + time. New filters with several inputs, that treat several frames at a time + or that require a special treatment at EOF should probably use the design + using activate(). + + activate + -------- + + This method is called when something must be done in a filter; the + definition of that "something" depends on the semantic of the filter. + + The callback must examine the status of the filter's links and proceed + accordingly. + + The status of output links is stored in the frame_wanted_out, status_in + and status_out fields and tested by the ff_outlink_frame_wanted() + function. If this function returns true, then the processing requires a + frame on this link and the filter is expected to make efforts in that + direction. + + The status of input links is stored by the status_in, fifo and + status_out fields; they must not be accessed directly. The fifo field + contains the frames that are queued in the input for processing by the + filter. The status_in and status_out fields contains the queued status + (EOF or error) of the link; status_in is a status change that must be + taken into account after all frames in fifo have been processed; + status_out is the status that have been taken into account, it is final + when it is not 0. + + The typical task of an activate callback is to first check the backward + status of output links, and if relevant forward it to the corresponding + input. Then, if relevant, for each input link: test the availability of + frames in fifo and process them; if no frame is available, test and + acknowledge a change of status using ff_inlink_acknowledge_status(); and + forward the result (frame or status change) to the corresponding input. + If nothing is possible, test the status of outputs and forward it to the + corresponding input(s). If still not possible, return FFERROR_NOT_READY. + + If the filters stores internally one or a few frame for some input, it + can consider them to be part of the FIFO and delay acknowledging a + status change accordingly. + + Example code: + + ret = ff_outlink_get_status(outlink); + if (ret) { + ff_inlink_set_status(inlink, ret); + return 0; + } + if (priv->next_frame) { + /* use it */ + return 0; + } + ret = ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + if (ret) { + /* use it */ + return 0; + } + ret = ff_inlink_acknowledge_status(inlink, &status, &pts); + if (ret) { + /* flush */ + ff_outlink_set_status(outlink, status, pts); + return 0; + } + if (ff_outlink_frame_wanted(outlink)) { + ff_inlink_request_frame(inlink); + return 0; + } + return FFERROR_NOT_READY; + + The exact code depends on how similar the /* use it */ blocks are and + how related they are to the /* flush */ block, and needs to apply these + operations to the correct inlink or outlink if there are several. + + Macros are available to factor that when no extra processing is needed: + + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + FF_FILTER_FORWARD_STATUS_ALL(outlink, filter); + FF_FILTER_FORWARD_STATUS(inlink, outlink); + FF_FILTER_FORWARD_STATUS_ALL(inlink, filter); + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + filter_frame + ------------ + + For filters that do not use the activate() callback, this method is + called when a frame is pushed to the filter's input. It can be called at + any time except in a reentrant way. + + If the input frame is enough to produce output, then the filter should + push the output frames on the output link immediately. + + As an exception to the previous rule, if the input frame is enough to + produce several output frames, then the filter needs output only at + least one per link. The additional frames can be left buffered in the + filter; these buffered frames must be flushed immediately if a new input + produces new output. + + (Example: frame rate-doubling filter: filter_frame must (1) flush the + second copy of the previous frame, if it is still there, (2) push the + first copy of the incoming frame, (3) keep the second copy for later.) + + If the input frame is not enough to produce output, the filter must not + call request_frame to get more. It must just process the frame or queue + it. The task of requesting more frames is left to the filter's + request_frame method or the application. + + If a filter has several inputs, the filter must be ready for frames + arriving randomly on any input. Therefore, any filter with several inputs + will most likely require some kind of queuing mechanism. It is perfectly + acceptable to have a limited queue and to drop frames when the inputs + are too unbalanced. + + request_frame + ------------- + + For filters that do not use the activate() callback, this method is + called when a frame is wanted on an output. + + For a source, it should directly call filter_frame on the corresponding + output. + + For a filter, if there are queued frames already ready, one of these + frames should be pushed. If not, the filter should request a frame on + one of its inputs, repeatedly until at least one frame has been pushed. + + Return values: + if request_frame could produce a frame, or at least make progress + towards producing a frame, it should return 0; + if it could not for temporary reasons, it should return AVERROR(EAGAIN); + if it could not because there are no more frames, it should return + AVERROR_EOF. + + The typical implementation of request_frame for a filter with several + inputs will look like that: + + if (frames_queued) { + push_one_frame(); + return 0; + } + input = input_where_a_frame_is_most_needed(); + ret = ff_request_frame(input); + if (ret == AVERROR_EOF) { + process_eof_on_input(); + } else if (ret < 0) { + return ret; + } + return 0; + + Note that, except for filters that can have queued frames and sources, + request_frame does not push frames: it requests them to its input, and + as a reaction, the filter_frame method possibly will be called and do + the work. |