Skip to content

Video Sequencer Editor

Code Layout

Main bulk of code is split in editor code and sequencer "core" code. In general, editor code should avoid accessing DNA fields directly and implementing own logic on these fields. This is because such code is usually shared with RNA functions and can miss calling necessary updates. Other related code is spread across codebase in RNA, transform code, dependency graph, scene functions, outliner and so on.

Sequencer core

File Use
source/blender/sequencer/intern/clipboard.c Provides copy/paste functionality.
source/blender/sequencer/intern/disk_cache.c Cache system extension that allows disk storage.
source/blender/sequencer/intern/effects.c Effect strips implementations.
source/blender/sequencer/intern/image_cache.c Cache system.
source/blender/sequencer/intern/iterator.c Strip Collection/Iterator.
source/blender/sequencer/intern/modifier.c Strip modifiers.
source/blender/sequencer/intern/multiview.c Utility functions for multiview images/movies.
source/blender/sequencer/intern/prefetch.c Prefetching system.
source/blender/sequencer/intern/proxy.c Proxy system.
source/blender/sequencer/intern/proxy_job.c Proxy system job.
source/blender/sequencer/intern/render.c Rendering pipeline and utility functions.
source/blender/sequencer/intern/sequence_lookup.c Strip name lookup cache, used to improve anmimation evaluation performance.
source/blender/sequencer/intern/sequencer.c Sequencer data management functions.
source/blender/sequencer/intern/sound.c Utility functions for sound strips.
source/blender/sequencer/intern/strip_add.c Functions for adding strips.
source/blender/sequencer/intern/strip_edit.c Functions for changing strip data or properties. These functions are meant to be used by strip editing operators.
source/blender/sequencer/intern/strip_relations.c Various functions handling relationship between strips such as recursivity checks, dependency checks and hierarchy based data invalidation.
source/blender/sequencer/intern/strip_select.c Functions for handling selection.
source/blender/sequencer/intern/strip_time.c Functions for getting / setting time properties and time related update functions.
source/blender/sequencer/intern/strip_transform.c Functions mainly used by transform operators. Not only for strip transformation, but also for image transformation.
source/blender/sequencer/intern/utils.c Functions that do not quite fit any category above.

Editor code

File Use
source/blender/editors/space_sequencer/sequencer_add.c Strip add operators.
source/blender/editors/space_sequencer/sequencer_buttons.c Side panel elements.
source/blender/editors/space_sequencer/sequencer_draw.c Drawing code for preview and timeline.
source/blender/editors/space_sequencer/sequencer_edit.c Strip editing operators.
source/blender/editors/space_sequencer/sequencer_modifier.c Operators for handling modifiers.
source/blender/editors/space_sequencer/sequencer_ops.c Registering operators.
source/blender/editors/space_sequencer/sequencer_preview.c Job for drawing waveforms in sound strips.
source/blender/editors/space_sequencer/sequencer_proxy.c Operators for proxy setup and building.
source/blender/editors/space_sequencer/sequencer_scopes.c Functions for image analysis.
source/blender/editors/space_sequencer/sequencer_select.c Operators for strip selection.
source/blender/editors/space_sequencer/sequencer_thumbnails.c Job for rendering thumbnails.
source/blender/editors/space_sequencer/sequencer_view.c Operators for handling editor view.
source/blender/editors/space_sequencer/space_sequencer.c Editor setup code.

Data storage

There are 2 main data structures containing sequencer data: - Editing holds data that belongs to sequencer core, this is mainly strips. - SpaceSeq holds data that belongs to sequencer editor. This defines editor state and configuration like render size, overlays and various view settings.

Editing itself belongs to Scene->ed. Most important field is seqbase which is linked list of all strips in timeline. metastack defines list of meta strips being edited. Runtime data for cache and prefetching are stored in Editing as well.

Sequencer also uses SequencerToolSettings which are stored in scene->toolsettings->sequencer_tool_settings

Strip data layout

This is quite confusing part of sequencer code - What is usually called a strip is represented by data structure called Sequence. There is also data structure with name Strip which defines data and some properties used by Sequence. Many fields in Sequence structure defines all properties of strip, mainly its position in timeline, type, properties for image processing or sound, pointers to other datablocks, effect data, etc.

Data structures nested in Sequence:

  • Strip: As mentioned above, this defines mainly file paths for image and movie strips along with some properties. These will be described below.
  • StripProxy: Defines proxies for image or movie strips. Proxy size, quality, storage methods and timecode(for movie strips only).
  • StripTransform: Image transformation properties.
  • StripCrop: Image crop properties.
  • StripElem: Source image or movie filename. In case of image sequences, array of this structure is used. Some basic information about media is stored here, such as image resolution or movie FPS. This information may not be accurate and should be used carefully.

Offsets:

  • startdisp / enddisp This indicates start and end frame of strip as it is truly presented in timeline. This is meant to be updated during editing. It is more of a runtime variable, but back then this wasn't differentiated by any means.
  • startstill / endstill indicates how many frames are there in strip before/after actual image content. Strip can be longer than video for example - area with no image content should preview as still frame.
  • startofs / endofs is offset from start of content. For example, to skip 200 frames from start, startofs should be set to 200. If this happens there can be no still frames, but this is not prevented by design, it's prevented by implementation. This offset is also called split or cut in other words.
  • anim_startofs / anim_endofs This is "hold split". If strip is hold split at 200 frames from start and start is extended, This will yield still frames.

Rendering

Sequencer uses mostly CPU to produce images (exception is text and scene strip type). Because of this, only very basic image processing is viable for real time preview and many functions are optimized to do minimum possible amount of work.

Rendering context

SeqRenderData data structure is used to set up how image should be rendered. Most important fields are rectx and recty to define image resolution of final image. Fields use_proxies with preview_render_size are useful to configure what proxy size should be used or to downscale image resolution for better performance. See ESpaceSeq_Proxy_RenderSize.

Strip stack rendering

Strips are rendered in sequence from bottom to top according to their channel (seq->machine). This is done in function seq_render_strip_stack() where strips are traversed from top to bottom first. If any strip can produce image on it's own, or it is stored in cache, traversing direction is reversed and strips are blended according to their blend mode. Result of strip stack rendering is always 1 image, which is stored in cache separately as final image.

Individual strip rendering

This is handled by function seq_render_strip(). First cache is checked if preprocessed or raw image exists. If no image is found in cache, function do_render_strip_uncached() dispatches rendering function based on strip type. This should give "raw" image which may not have correct resolution and does not have any image processing applied like modifiers or transformation. To apply required image processing, seq_render_preprocess_ibuf() is used. If no processing is required, image is returned as-is.

Preprocessing stage

This stage is mainly used to apply corrections on original image. This includes:

  • Deinterlacing
  • Transformation and cropping
  • Flipping
  • Adjusting saturation
  • Byte -> float conversion
  • Pixel value multiplication
  • Modifiers

Due to sequencer design, this stage is also used, when image size does not match rendering context size. This is because it can be done during transformation along with user defined properties at once and therefore much faster than doing these steps separately.

Note: Effect strips can also use preprocessing stage. Strips that support Multiviev rendering need to apply preprocessing stage for each rendered image.

Cache

During rendering, images are stored in cache, to allow for faster redrawing when they are viewed repeatedly. Images are stored at various stages of processing, so this is also beneficial when tweaking properties of strips. Cache is also used for storage when prefetching is enabled.

RAM is normally used for storage, but disk cache is implemented to allow storage with greater capacity at expense of performance. Disk cache is designed to be extension of RAM cache on implementation level as well.

Image caching can be configured globally or per strip with developer extras enabled. It is also possible to enable cache overlay in timeline(view->cache menu) to visualize cached(RAM only) images in time and per strip. This is useful for debugging not only cache but also to ensure consistency when doing rendering performance tests.

Stages stored in cache:

  • Raw images: SEQ_CACHE_STORE_RAW.
  • Peprocessed images: SEQ_CACHE_STORE_PREPROCESSED.
  • Composited (2 strips blended together) images: SEQ_CACHE_STORE_COMPOSITE.
  • Final images (last blended image): SEQ_CACHE_STORE_FINAL_OUT. This image is copy of SEQ_CACHE_STORE_COMPOSITE stage and takes no additional memory to be stored.

These are all stages in which image can be changed, however change in lower level will influence all subsequent images. This is handled on 2 levels: By cache invalidation functions, which traverse strips and invalidate(delete) images of dependant strips. Cache also links images within each frame, which helps with invalidation, but it should not be necessary.

RAM cache design

Sequencer cache was originally using MovieCache, which works well with single strip, but it is quite limited otherwise. Main limitation for sequencer is memory usage limiting mechanism, which does free items on simple criteria like their age or distance from current frame. In sequencer however, there are more items that meet this condition and random items are freed. To further complicate this situation, not all images have same size in memory, which means that sometimes image can fail to be cached resulting in uneven caching with random images missing.

This is resolved by 2 changes:

  • Memory limit is not strictly obeyed. All images created during rendering are added to cache, even if the cache is already full. Only before caching SEQ_CACHE_STORE_FINAL_OUT, memory is freed.
  • Cached images for each frame are linked, which makes it possible to free them all at once, even if some of strips use speed effect which causes rendering of different frame number.

To conserve memory usage, not all cached stages should be stored for longer term. Currently SEQ_CACHE_STORE_PREPROCESSED and SEQ_CACHE_STORE_COMPOSITE are marked as temporary, which means they will be freed when next frame is rendered. This still allows these stages to be useful when tweaking strip property and frame does not change for example.

Another limitation was introduction of prefetching system, which works by filling RAM cache as much as possible in advance of current frame. For this to work reliably, change in image freeing priority must change from furthest cached frame to "ideally leftmost cached frame outside of prefetching area, otherwise rightmost frame outside of prefetching area".

Latest addition to cache is thumbnail storage, which is rather hacked-in and ideally will be isolated in future. This cache "stage" or type is very similar to SEQ_CACHE_STORE_RAW in that it is valid for all strips that share same path to file. Ideally new cache system for such entries can be created which can be compatible with disk cache and cache invalidation mechanisms.

Disk cache design

Disk cache is extension of RAM cache. Storage is configured in user preferences.

Main considerations for disk cache:

  • Images are stored pernamently until invalidated or removed due to disk space limits.
  • Multiple images are stored in 1 file to reduce cold read delays.
  • It must be possible to write images in random order, because rendering is not always sequential.
  • 2 blend files should never use same disk cache path.
  • It should support most features as RAM cache, for example float images. Images must be recovered without any quality loss.

To write images at random order, each cache file has header that maps file contents. This way file can always be appended.

To ensure maximum possible integrity of cache, file name does contain values used by RAM cache to calculate hash, but these values are human readable. Path to cache files consists of blend file name, scene name and RAM cache creation timestamp and strip name. Example of typical path: untitled.blend_seq_cache/SCScene-1638169261/SQtime.002/8-1920x1080-99%(0)-1.dcf

To future-proof the design, versioning system is implemented. Each blend file cache directory has cache_version file. If version is not matched, all files in cache directory are removed.

Disk cache does support image compression with standard compression algorithms. During testing, these seems to be more effective than image codecs, especially lossless ones.

To avoid unnecessary disk IO, cache directory structure is traversed and DiskCacheFile structure representing each file is stored in RAM. Every new cache file that is created is stored in RAM as well. This helps mainly when cache files are invalidated, so whole tree does not have to be traversed possibly during rendering.

Iterator

To make work with sets of strips easier, iterator system can be used. Main design goal is to provide means to inline logic, instead of creating callback functions. SeqCollection structure defines a set of strips and can be iterated on by SEQ_ITERATOR_FOREACH macro, similar to LISTBASE_FOREACH macro. To create SeqCollection, most common queries are available, but custom functions can be created. Use of this iterator system is encouraged everywhere where maximum possible performance is not essential. Because SeqCollection can be reused multiple times and it can be passed as argument, it may be faster than looping over strips multiple times.

For cases where logic may be complicated such as working on strips not only selected by simple criteria, but also their relationships, there are functions for combining, filtering and expanding SeqCollection. This way a bulk of logic responsible for selecting strips that are operated on can be separated from functional logic which does change their data.

Implementation should be well documented, but good example is for example select_grouped_effect_link():

/* Select all strips within time range and with lower channel of initial selection. Then select  
 * effect chains of these strips. */  
static bool select_grouped_effect_link(SeqCollection *strips,  
                                       ListBase *seqbase,  
                                       Sequence *UNUSED(actseq),  
                                       const int UNUSED(channel))  
{  
  /* Get collection of strips. */  
  SEQ_filter_selected_strips(strips);  
  const int selected_strip_count = SEQ_collection_len(strips);  
  SEQ_collection_expand(seqbase, strips, query_lower_channel_strips);  
  SEQ_collection_expand(seqbase, strips, SEQ_query_strip_effect_chain);  

  /* Check if other strips will be affected. */  
  const bool changed = SEQ_collection_len(strips) > selected_strip_count;  

  /* Actual logic. */  
  Sequence *seq;  
  SEQ_ITERATOR_FOREACH (seq, strips) {  
    seq->flag |= SELECT;  
  }  

  return changed;  
}

This function is supplied with collection of strips, contents depend on whether operator is executed from preview where only visible strips are considered, or timeline, in which case all strips will be included. First collection is filtered to only selected strips. Number of strips is saved for later comparison to check if this operation had any effect. Function SEQ_collection_expand() is used to expand original collection. In practice this means that each strip in sequencer is compared to each strip in provided collection. Callback query function then does selection based on its internal logic. in case of query_lower_channel_strips(), strip is selected if it is visually under strips from provided collection. in similar fashion, function SEQ_query_strip_effect_chain selects every strip, that has effect relationship to strips in provided collection. These are quite complicated selection criteria, that would be hard to parse if done by plain LISTBASE_FOREACH iterations. Finally collection is iterated and operator logic is executed - strips are selected.

Note, implementation of this design may change a bit, for example SEQ_query_strip_effect_chain may become standalone function, that will take SeqCollection as argument. Function SEQ_collection_expand() is used mainly because it does include logic to prevent infinite loops which are likely in more complicated queries.

Proxy system

To optimize video files playback performance, sequencer uses proxy system which is available in movie clip editor too. Proxies are re-encoded video files with settings to get good scrubbing performance and they can have smaller resolution. Since proxy transcoding is standalone system, its inner workings won't be described in detail here, but rather its integration with sequencer. Same proxy system is used in movie clip editor.

TODO:

  • Image strips
  • Generating editor side, sequencer side
  • How proxies are used

Prefetching

Prefetching can be used to improve consistency of playback speed if image processing demands are unevenly spread. This is done by rendering frames in advance in background job and filling cache up to its limit. This way overhead of drawing code is eliminated too, even though it's usually not significant.

Startup and stop procedure

When prefetching is enabled, function SEQ_render_give_ibuf() which is responsible for rendering timeline content will try to start prefetching by calling function seq_prefetch_start. If prefetch thread is running, but it is suspended, it is resumed. On any update done to sequencer data that affects image output, prefetch must be stopped by calling SEQ_prefetch_stop(). This is done as a part of cache invalidation, but there are cases when it has to be done explicitly, for example on undo. When image is re-rendered after change, prefetching will restart with new data.

Rendering loop

Because DNA structures may contain runtime data necessary for operation, prefetch operates on Scene copy, which contains sequencer data as well. It hosts its own dependency graph to evaluate data for each frame and a copy of SeqRenderData rendering context. This context is marked as for prefetching(is_prefetch_render), so another prefetching job is not created from prefetching thread. Finally function seq_prefetch_frames() is executed in a thread. Rendering loop uses dependency loop to evaluate all datablocks. There is call to BKE_animsys_evaluate_animdata which I am not entirely sure why it is needed as it shouldn't be. Function SEQ_render_give_ibuf() is called on copy of scene and rendering context. Rendering loop can be suspended by timeline scrubbing or cache being full. When there are no more frames to render or prefetching is disabled, job is terminated. Loop resumes or starts, when cache is freed again.

Cache handling

Prefetching job has no prior information on where it should stop. This is because each frame may have different size in memory or frames may be missing. Further it does not know about total memory usage since it may change outside of sequencer. Because of this function seq_prefetch_do_suspend() polls function seq_prefetch_is_cache_full() on every iteration of rendering loop. If cache is full, thread will be suspended. Once cache is freed through startup procedure signal is sent to prefetching thread to check memory usage and resume render loop if possible. Unlike normal sequencer rendering procedure, prefetch job must store SEQ_CACHE_STORE_FINAL_OUT image regardless of memory usage. This is, because function seq_cache_put_if_possible would not store image if it's size is greater than available memory and therefore function seq_prefetch_is_cache_full() would never return true. Prefetch job relies on disregarding this limit, but since this will cause thread being suspended, it will never be crossed by large amount.

Another important fact is, that each scene with sequencer data has it's own cache. This means, that calling function SEQ_render_give_ibuf() with copy of original data would store image with different context and it wouldn't be possible to get it from context that is used to draw UI. This is resolved by referencing original prefetch job in copied scene - pfjob->scene_eval->ed->prefetch_job = pfjob; In cache code original data can be accessed with function seq_prefetch_get_original_context(). This is not great way to resolve the issue though.

Finally, cache freeing strategy is changed from furthest cached frame to "ideally leftmost cached frame outside of prefetching area, otherwise rightmost frame outside of prefetching area". This is, because user can move current frame few frames before, inside or after prefetched area. This can also happen frequently, so freeing strategy should be least disruptive and preserve continuous flow of cached images as much as it can.

To debug prefetching process, cache overlay is useful tool. To continuously update cache overlay, function SEQ_prefetch_need_redraw is polled from sequencer drawing code and adds notifier to redraw itself which creates event loop. Event loop is terminated when prefetching is suspended.

Limitations

Currently 3D scene strips can't be prefetched. This is for 2 reasons - They use offscreen rendering, which requires interlock with main frame, which is not implemented. Even if background rendering would be used, problem is that dependency graph used by prefetch job can't be used to create dependency graph for render engine.

Thumbnails

TODO