Video Sequencer Editor¶
Code Layout¶
Main bulk of code is split in editor code and sequencer "core" code. In general, editor code should avoid accessing DNA fields directly and implementing own logic on these fields. This is because such code is usually shared with RNA functions and can miss calling necessary updates. Other related code is spread across codebase in RNA, transform code, dependency graph, scene functions, outliner and so on.
Sequencer core¶
File | Use |
---|---|
source/blender/sequencer/intern/clipboard.c | Provides copy/paste functionality. |
source/blender/sequencer/intern/disk_cache.c | Cache system extension that allows disk storage. |
source/blender/sequencer/intern/effects.c | Effect strips implementations. |
source/blender/sequencer/intern/image_cache.c | Cache system. |
source/blender/sequencer/intern/iterator.c | Strip Collection/Iterator. |
source/blender/sequencer/intern/modifier.c | Strip modifiers. |
source/blender/sequencer/intern/multiview.c | Utility functions for multiview images/movies. |
source/blender/sequencer/intern/prefetch.c | Prefetching system. |
source/blender/sequencer/intern/proxy.c | Proxy system. |
source/blender/sequencer/intern/proxy_job.c | Proxy system job. |
source/blender/sequencer/intern/render.c | Rendering pipeline and utility functions. |
source/blender/sequencer/intern/sequence_lookup.c | Strip name lookup cache, used to improve anmimation evaluation performance. |
source/blender/sequencer/intern/sequencer.c | Sequencer data management functions. |
source/blender/sequencer/intern/sound.c | Utility functions for sound strips. |
source/blender/sequencer/intern/strip_add.c | Functions for adding strips. |
source/blender/sequencer/intern/strip_edit.c | Functions for changing strip data or properties. These functions are meant to be used by strip editing operators. |
source/blender/sequencer/intern/strip_relations.c | Various functions handling relationship between strips such as recursivity checks, dependency checks and hierarchy based data invalidation. |
source/blender/sequencer/intern/strip_select.c | Functions for handling selection. |
source/blender/sequencer/intern/strip_time.c | Functions for getting / setting time properties and time related update functions. |
source/blender/sequencer/intern/strip_transform.c | Functions mainly used by transform operators. Not only for strip transformation, but also for image transformation. |
source/blender/sequencer/intern/utils.c | Functions that do not quite fit any category above. |
Editor code¶
File | Use |
---|---|
source/blender/editors/space_sequencer/sequencer_add.c | Strip add operators. |
source/blender/editors/space_sequencer/sequencer_buttons.c | Side panel elements. |
source/blender/editors/space_sequencer/sequencer_draw.c | Drawing code for preview and timeline. |
source/blender/editors/space_sequencer/sequencer_edit.c | Strip editing operators. |
source/blender/editors/space_sequencer/sequencer_modifier.c | Operators for handling modifiers. |
source/blender/editors/space_sequencer/sequencer_ops.c | Registering operators. |
source/blender/editors/space_sequencer/sequencer_preview.c | Job for drawing waveforms in sound strips. |
source/blender/editors/space_sequencer/sequencer_proxy.c | Operators for proxy setup and building. |
source/blender/editors/space_sequencer/sequencer_scopes.c | Functions for image analysis. |
source/blender/editors/space_sequencer/sequencer_select.c | Operators for strip selection. |
source/blender/editors/space_sequencer/sequencer_thumbnails.c | Job for rendering thumbnails. |
source/blender/editors/space_sequencer/sequencer_view.c | Operators for handling editor view. |
source/blender/editors/space_sequencer/space_sequencer.c | Editor setup code. |
Data storage¶
There are 2 main data structures containing sequencer data: -
Editing
holds data that belongs to sequencer core, this is mainly
strips. - SpaceSeq
holds data that belongs to sequencer editor. This
defines editor state and configuration like render size, overlays and
various view settings.
Editing
itself belongs to Scene->ed
. Most important field is
seqbase
which is linked list of all strips in timeline.
metastack
defines list of meta strips being edited. Runtime data for
cache and prefetching are stored in Editing
as well.
Sequencer also uses SequencerToolSettings
which are stored in
scene->toolsettings->sequencer_tool_settings
Strip data layout¶
This is quite confusing part of sequencer code - What is usually called
a strip is represented by data structure called Sequence
. There is
also data structure with name Strip
which defines data and some
properties used by Sequence
. Many fields in Sequence
structure
defines all properties of strip, mainly its position in timeline, type,
properties for image processing or sound, pointers to other datablocks,
effect data, etc.
Data structures nested in Sequence
:
Strip
: As mentioned above, this defines mainly file paths for image and movie strips along with some properties. These will be described below.StripProxy
: Defines proxies for image or movie strips. Proxy size, quality, storage methods and timecode(for movie strips only).StripTransform
: Image transformation properties.StripCrop
: Image crop properties.StripElem
: Source image or movie filename. In case of image sequences, array of this structure is used. Some basic information about media is stored here, such as image resolution or movie FPS. This information may not be accurate and should be used carefully.
Offsets:
startdisp
/enddisp
This indicates start and end frame of strip as it is truly presented in timeline. This is meant to be updated during editing. It is more of a runtime variable, but back then this wasn't differentiated by any means.startstill
/endstill
indicates how many frames are there in strip before/after actual image content. Strip can be longer than video for example - area with no image content should preview as still frame.startofs
/endofs
is offset from start of content. For example, to skip 200 frames from start,startofs
should be set to200
. If this happens there can be no still frames, but this is not prevented by design, it's prevented by implementation. This offset is also called split or cut in other words.anim_startofs
/anim_endofs
This is "hold split". If strip is hold split at 200 frames from start and start is extended, This will yield still frames.
Rendering¶
Sequencer uses mostly CPU to produce images (exception is text and scene strip type). Because of this, only very basic image processing is viable for real time preview and many functions are optimized to do minimum possible amount of work.
Rendering context¶
SeqRenderData
data structure is used to set up how image should be
rendered. Most important fields are rectx
and recty
to define
image resolution of final image. Fields use_proxies
with
preview_render_size
are useful to configure what proxy size should
be used or to downscale image resolution for better performance. See
ESpaceSeq_Proxy_RenderSize
.
Strip stack rendering¶
Strips are rendered in sequence from bottom to top according to their
channel (seq->machine
). This is done in function
seq_render_strip_stack()
where strips are traversed from top to
bottom first. If any strip can produce image on it's own, or it is
stored in cache, traversing direction is reversed and strips are blended
according to their blend mode. Result of strip stack rendering is always
1 image, which is stored in cache separately as final image.
Individual strip rendering¶
This is handled by function seq_render_strip()
. First cache is
checked if preprocessed or raw image exists. If no image is found in
cache, function do_render_strip_uncached()
dispatches rendering
function based on strip type. This should give "raw" image which may not
have correct resolution and does not have any image processing applied
like modifiers or transformation. To apply required image processing,
seq_render_preprocess_ibuf()
is used. If no processing is required,
image is returned as-is.
Preprocessing stage¶
This stage is mainly used to apply corrections on original image. This includes:
- Deinterlacing
- Transformation and cropping
- Flipping
- Adjusting saturation
- Byte -> float conversion
- Pixel value multiplication
- Modifiers
Due to sequencer design, this stage is also used, when image size does not match rendering context size. This is because it can be done during transformation along with user defined properties at once and therefore much faster than doing these steps separately.
Note: Effect strips can also use preprocessing stage. Strips that support Multiviev rendering need to apply preprocessing stage for each rendered image.
Cache¶
During rendering, images are stored in cache, to allow for faster redrawing when they are viewed repeatedly. Images are stored at various stages of processing, so this is also beneficial when tweaking properties of strips. Cache is also used for storage when prefetching is enabled.
RAM is normally used for storage, but disk cache is implemented to allow storage with greater capacity at expense of performance. Disk cache is designed to be extension of RAM cache on implementation level as well.
Image caching can be configured globally or per strip with developer extras enabled. It is also possible to enable cache overlay in timeline(view->cache menu) to visualize cached(RAM only) images in time and per strip. This is useful for debugging not only cache but also to ensure consistency when doing rendering performance tests.
Stages stored in cache:
- Raw images:
SEQ_CACHE_STORE_RAW
. - Peprocessed images:
SEQ_CACHE_STORE_PREPROCESSED
. - Composited (2 strips blended together) images:
SEQ_CACHE_STORE_COMPOSITE
. - Final images (last blended image):
SEQ_CACHE_STORE_FINAL_OUT
. This image is copy ofSEQ_CACHE_STORE_COMPOSITE
stage and takes no additional memory to be stored.
These are all stages in which image can be changed, however change in lower level will influence all subsequent images. This is handled on 2 levels: By cache invalidation functions, which traverse strips and invalidate(delete) images of dependant strips. Cache also links images within each frame, which helps with invalidation, but it should not be necessary.
RAM cache design¶
Sequencer cache was originally using MovieCache
, which works well
with single strip, but it is quite limited otherwise. Main limitation
for sequencer is memory usage limiting mechanism, which does free items
on simple criteria like their age or distance from current frame. In
sequencer however, there are more items that meet this condition and
random items are freed. To further complicate this situation, not all
images have same size in memory, which means that sometimes image can
fail to be cached resulting in uneven caching with random images
missing.
This is resolved by 2 changes:
- Memory limit is not strictly obeyed. All images created during rendering are added to cache, even if the cache is already full. Only before caching SEQ_CACHE_STORE_FINAL_OUT, memory is freed.
- Cached images for each frame are linked, which makes it possible to free them all at once, even if some of strips use speed effect which causes rendering of different frame number.
To conserve memory usage, not all cached stages should be stored for
longer term. Currently SEQ_CACHE_STORE_PREPROCESSED
and
SEQ_CACHE_STORE_COMPOSITE
are marked as temporary, which means they
will be freed when next frame is rendered. This still allows these
stages to be useful when tweaking strip property and frame does not
change for example.
Another limitation was introduction of prefetching system, which works by filling RAM cache as much as possible in advance of current frame. For this to work reliably, change in image freeing priority must change from furthest cached frame to "ideally leftmost cached frame outside of prefetching area, otherwise rightmost frame outside of prefetching area".
Latest addition to cache is thumbnail storage, which is rather hacked-in
and ideally will be isolated in future. This cache "stage" or type is
very similar to SEQ_CACHE_STORE_RAW
in that it is valid for all
strips that share same path to file. Ideally new cache system for such
entries can be created which can be compatible with disk cache and cache
invalidation mechanisms.
Disk cache design¶
Disk cache is extension of RAM cache. Storage is configured in user preferences.
Main considerations for disk cache:
- Images are stored pernamently until invalidated or removed due to disk space limits.
- Multiple images are stored in 1 file to reduce cold read delays.
- It must be possible to write images in random order, because rendering is not always sequential.
- 2 blend files should never use same disk cache path.
- It should support most features as RAM cache, for example float images. Images must be recovered without any quality loss.
To write images at random order, each cache file has header that maps file contents. This way file can always be appended.
To ensure maximum possible integrity of cache, file name does contain
values used by RAM cache to calculate hash, but these values are human
readable. Path to cache files consists of blend file name, scene name
and RAM cache creation timestamp and strip name. Example of typical
path:
untitled.blend_seq_cache/SCScene-1638169261/SQtime.002/8-1920x1080-99%(0)-1.dcf
To future-proof the design, versioning system is implemented. Each blend
file cache directory has cache_version
file. If version is not
matched, all files in cache directory are removed.
Disk cache does support image compression with standard compression algorithms. During testing, these seems to be more effective than image codecs, especially lossless ones.
To avoid unnecessary disk IO, cache directory structure is traversed and
DiskCacheFile
structure representing each file is stored in RAM.
Every new cache file that is created is stored in RAM as well. This
helps mainly when cache files are invalidated, so whole tree does not
have to be traversed possibly during rendering.
Iterator¶
To make work with sets of strips easier, iterator system can be used.
Main design goal is to provide means to inline logic, instead of
creating callback functions. SeqCollection
structure defines a set
of strips and can be iterated on by SEQ_ITERATOR_FOREACH
macro,
similar to LISTBASE_FOREACH
macro. To create SeqCollection
, most
common queries are available, but custom functions can be created. Use
of this iterator system is encouraged everywhere where maximum possible
performance is not essential. Because SeqCollection
can be reused
multiple times and it can be passed as argument, it may be faster than
looping over strips multiple times.
For cases where logic may be complicated such as working on strips not
only selected by simple criteria, but also their relationships, there
are functions for combining, filtering and expanding SeqCollection
.
This way a bulk of logic responsible for selecting strips that are
operated on can be separated from functional logic which does change
their data.
Implementation should be well documented, but good example is for
example select_grouped_effect_link()
:
/* Select all strips within time range and with lower channel of initial selection. Then select
* effect chains of these strips. */
static bool select_grouped_effect_link(SeqCollection *strips,
ListBase *seqbase,
Sequence *UNUSED(actseq),
const int UNUSED(channel))
{
/* Get collection of strips. */
SEQ_filter_selected_strips(strips);
const int selected_strip_count = SEQ_collection_len(strips);
SEQ_collection_expand(seqbase, strips, query_lower_channel_strips);
SEQ_collection_expand(seqbase, strips, SEQ_query_strip_effect_chain);
/* Check if other strips will be affected. */
const bool changed = SEQ_collection_len(strips) > selected_strip_count;
/* Actual logic. */
Sequence *seq;
SEQ_ITERATOR_FOREACH (seq, strips) {
seq->flag |= SELECT;
}
return changed;
}
This function is supplied with collection of strips, contents depend on
whether operator is executed from preview where only visible strips are
considered, or timeline, in which case all strips will be included.
First collection is filtered to only selected strips. Number of strips
is saved for later comparison to check if this operation had any effect.
Function SEQ_collection_expand()
is used to expand original
collection. In practice this means that each strip in sequencer is
compared to each strip in provided collection. Callback query function
then does selection based on its internal logic. in case of
query_lower_channel_strips()
, strip is selected if it is visually
under strips from provided collection. in similar fashion, function
SEQ_query_strip_effect_chain
selects every strip, that has effect
relationship to strips in provided collection. These are quite
complicated selection criteria, that would be hard to parse if done by
plain LISTBASE_FOREACH
iterations. Finally collection is iterated
and operator logic is executed - strips are selected.
Note, implementation of this design may change a bit, for example
SEQ_query_strip_effect_chain
may become standalone function, that
will take SeqCollection
as argument. Function
SEQ_collection_expand()
is used mainly because it does include logic
to prevent infinite loops which are likely in more complicated queries.
Proxy system¶
To optimize video files playback performance, sequencer uses proxy system which is available in movie clip editor too. Proxies are re-encoded video files with settings to get good scrubbing performance and they can have smaller resolution. Since proxy transcoding is standalone system, its inner workings won't be described in detail here, but rather its integration with sequencer. Same proxy system is used in movie clip editor.
TODO:
- Image strips
- Generating editor side, sequencer side
- How proxies are used
Prefetching¶
Prefetching can be used to improve consistency of playback speed if image processing demands are unevenly spread. This is done by rendering frames in advance in background job and filling cache up to its limit. This way overhead of drawing code is eliminated too, even though it's usually not significant.
Startup and stop procedure¶
When prefetching is enabled, function SEQ_render_give_ibuf()
which
is responsible for rendering timeline content will try to start
prefetching by calling function seq_prefetch_start
. If prefetch
thread is running, but it is suspended, it is resumed. On any update
done to sequencer data that affects image output, prefetch must be
stopped by calling SEQ_prefetch_stop()
. This is done as a part of
cache invalidation, but there are cases when it has to be done
explicitly, for example on undo. When image is re-rendered after change,
prefetching will restart with new data.
Rendering loop¶
Because DNA structures may contain runtime data necessary for operation,
prefetch operates on Scene
copy, which contains sequencer data as
well. It hosts its own dependency graph to evaluate data for each frame
and a copy of SeqRenderData
rendering context. This context is
marked as for prefetching(is_prefetch_render
), so another
prefetching job is not created from prefetching thread. Finally function
seq_prefetch_frames()
is executed in a thread. Rendering loop uses
dependency loop to evaluate all datablocks. There is call to
BKE_animsys_evaluate_animdata
which I am not entirely sure why it is
needed as it shouldn't be. Function SEQ_render_give_ibuf()
is called
on copy of scene and rendering context. Rendering loop can be suspended
by timeline scrubbing or cache being full. When there are no more frames
to render or prefetching is disabled, job is terminated. Loop resumes or
starts, when cache is freed again.
Cache handling¶
Prefetching job has no prior information on where it should stop. This
is because each frame may have different size in memory or frames may be
missing. Further it does not know about total memory usage since it may
change outside of sequencer. Because of this function
seq_prefetch_do_suspend()
polls function
seq_prefetch_is_cache_full()
on every iteration of rendering loop.
If cache is full, thread will be suspended. Once cache is freed through
startup procedure signal is sent to prefetching thread to check memory
usage and resume render loop if possible. Unlike normal sequencer
rendering procedure, prefetch job must store
SEQ_CACHE_STORE_FINAL_OUT
image regardless of memory usage. This is,
because function seq_cache_put_if_possible
would not store image if
it's size is greater than available memory and therefore function
seq_prefetch_is_cache_full()
would never return true
. Prefetch
job relies on disregarding this limit, but since this will cause thread
being suspended, it will never be crossed by large amount.
Another important fact is, that each scene with sequencer data has it's
own cache. This means, that calling function SEQ_render_give_ibuf()
with copy of original data would store image with different context and
it wouldn't be possible to get it from context that is used to draw UI.
This is resolved by referencing original prefetch job in copied scene -
pfjob->scene_eval->ed->prefetch_job = pfjob;
In cache code
original data can be accessed with function
seq_prefetch_get_original_context()
. This is not great way to
resolve the issue though.
Finally, cache freeing strategy is changed from furthest cached frame to "ideally leftmost cached frame outside of prefetching area, otherwise rightmost frame outside of prefetching area". This is, because user can move current frame few frames before, inside or after prefetched area. This can also happen frequently, so freeing strategy should be least disruptive and preserve continuous flow of cached images as much as it can.
To debug prefetching process, cache overlay is useful tool. To
continuously update cache overlay, function SEQ_prefetch_need_redraw
is polled from sequencer drawing code and adds notifier to redraw itself
which creates event loop. Event loop is terminated when prefetching is
suspended.
Limitations¶
Currently 3D scene strips can't be prefetched. This is for 2 reasons - They use offscreen rendering, which requires interlock with main frame, which is not implemented. Even if background rendering would be used, problem is that dependency graph used by prefetch job can't be used to create dependency graph for render engine.
Thumbnails¶
TODO