Source/VSE

= Video Sequencer Editor =

Code Layout
Main bulk of code is split in editor code and sequencer "core" code. In general, editor code should avoid accessing DNA fields directly and implementing own logic on these fields. This is because such code is usually shared with RNA functions and can miss calling necessary updates. Other related code is spread across codebase in RNA, transform code, dependency graph, scene functions, outliner and so on.

Data storage
There are 2 main data structures containing sequencer data: - `Editing` holds data that belongs to sequencer core, this is mainly strips. - `SpaceSeq` holds data that belongs to sequencer editor. This defines editor state and configuration like render size, overlays and various view settings.

`Editing` itself belongs to `Scene->ed`. Most important field is `seqbase` which is linked list of all strips in timeline. `metastack` defines list of meta strips being edited. Runtime data for cache and prefetching are stored in `Editing` as well.

Sequencer also uses `SequencerToolSettings` which are stored in `scene->toolsettings->sequencer_tool_settings`

Strip data layout
This is quite confusing part of sequencer code - What is usually called a strip is represented by data structure called `Sequence`. There is also data structure with name `Strip` which defines data and some properties used by `Sequence`. Many fields in `Sequence` structure defines all properties of strip, mainly its position in timeline, type, properties for image processing or sound, pointers to other datablocks, effect data, etc.

Data structures nested in `Sequence`:
 * `Strip`: As mentioned above, this defines mainly file paths for image and movie strips along with some properties. These will be described below.
 * `StripProxy`: Defines proxies for image or movie strips. Proxy size, quality, storage methods and timecode(for movie strips only).
 * `StripTransform`: Image transformation properties.
 * `StripCrop`: Image crop properties.
 * `StripElem`: Source image or movie filename. In case of image sequences, array of this structure is used. Some basic information about media is stored here, such as image resolution or movie FPS. This information may not be accurate and should be used carefully.

Offsets:
 * `startdisp` / `enddisp` This indicates start and end frame of strip as it is truly presented in timeline. This is meant to be updated during editing. It is more of a runtime variable, but back then this wasn't differentiated by any means.
 * `startstill` / `endstill` indicates how many frames are there in strip before/after actual image content. Strip can be longer than video for example - area with no image content should preview as still frame.
 * `startofs` / `endofs` is offset from start of content. For example, to skip 200 frames from start, `startofs` should be set to `200`. If this happens there can be no still frames, but this is not prevented by design, it's prevented by implementation. This offset is also called split or cut in other words.
 * `anim_startofs` / `anim_endofs` This is "hold split". If strip is hold split at 200 frames from start and start is extended, This will yield still frames.

Rendering
Sequencer uses mostly CPU to produce images (exception is text and scene strip type). Because of this, only very basic image processing is viable for real time preview and many functions are optimized to do minimum possible amount of work.

Rendering context
`SeqRenderData` data structure is used to set up how image should be rendered. Most important fields are `rectx` and `recty` to define image resolution of final image. Fields `use_proxies` with `preview_render_size` are useful to configure what proxy size should be used or to downscale image resolution for better performance. See `ESpaceSeq_Proxy_RenderSize`.

Strip stack rendering
Strips are rendered in sequence from bottom to top according to their channel (`seq->machine`). This is done in function `seq_render_strip_stack` where strips are traversed from top to bottom first. If any strip can produce image on it's own, or it is stored in cache, traversing direction is reversed and strips are blended according to their blend mode. Result of strip stack rendering is always 1 image, which is stored in cache separately as final image.

Individual strip rendering
This is handled by function `seq_render_strip`. First cache is checked if preprocessed or raw image exists. If no image is found in cache, function `do_render_strip_uncached` dispatches rendering function based on strip type. This should give "raw" image which may not have correct resolution and does not have any image processing applied like modifiers or transformation. To apply required image processing, `seq_render_preprocess_ibuf` is used. If no processing is required, image is returned as-is.

Preprocessing stage
This stage is mainly used to apply corrections on original image. This includes: Due to sequencer design, this stage is also used, when image size does not match rendering context size. This is because it can be done during transformation along with user defined properties at once and therefore much faster than doing these steps separately.
 * Deinterlacing
 * Transformation and cropping
 * Flipping
 * Adjusting saturation
 * Byte -> float conversion
 * Pixel value multiplication
 * Modifiers

Note: Effect strips can also use preprocessing stage. Strips that support Multiviev rendering need to apply preprocessing stage for each rendered image.

Cache
During rendering, images are stored in cache, to allow for faster redrawing when they are viewed repeatedly. Images are stored at various stages of processing, so this is also beneficial when tweaking properties of strips. Cache is also used for storage when prefetching is enabled.

RAM is normally used for storage, but disk cache is implemented to allow storage with greater capacity at expense of performance. Disk cache is designed to be extension of RAM cache on implementation level as well.

Image caching can be configured globally or per strip with developer extras enabled. It is also possible to enable cache overlay in timeline(view->cache menu) to visualize cached(RAM only) images in time and per strip. This is useful for debugging not only cache but also to ensure consistency when doing rendering performance tests.

Stages stored in cache: These are all stages in which image can be changed, however change in lower level will influence all subsequent images. This is handled on 2 levels: By cache invalidation functions, which traverse strips and invalidate(delete) images of dependant strips. Cache also links images within each frame, which helps with invalidation, but it should not be necessary.
 * Raw images: `SEQ_CACHE_STORE_RAW`.
 * Peprocessed images: `SEQ_CACHE_STORE_PREPROCESSED`.
 * Composited (2 strips blended together) images: `SEQ_CACHE_STORE_COMPOSITE`.
 * Final images (last blended image): `SEQ_CACHE_STORE_FINAL_OUT`. This image is copy of `SEQ_CACHE_STORE_COMPOSITE` stage and takes no additional memory to be stored.

RAM cache design
Sequencer cache was originally using `MovieCache`, which works well with single strip, but it is quite limited otherwise. Main limitation for sequencer is memory usage limiting mechanism, which does free items on simple criteria like their age or distance from current frame. In sequencer however, there are more items that meet this condition and random items are freed. To further complicate this situation, not all images have same size in memory, which means that sometimes image can fail to be cached resulting in uneven caching with random images missing.

This is resolved by 2 changes:
 * Memory limit is not strictly obeyed. All images created during rendering are added to cache, even if the cache is already full. Only before caching SEQ_CACHE_STORE_FINAL_OUT, memory is freed.
 * Cached images for each frame are linked, which makes it possible to free them all at once, even if some of strips use speed effect which causes rendering of different frame number.

To conserve memory usage, not all cached stages should be stored for longer term. Currently `SEQ_CACHE_STORE_PREPROCESSED` and `SEQ_CACHE_STORE_COMPOSITE` are marked as temporary, which means they will be freed when next frame is rendered. This still allows these stages to be useful when tweaking strip property and frame does not change for example.

Another limitation was introduction of prefetching system, which works by filling RAM cache as much as possible in advance of current frame. For this to work reliably, change in image freeing priority must change from furthest cached frame to "ideally leftmost cached frame outside of prefetching area, otherwise rightmost frame outside of prefetching area".

Latest addition to cache is thumbnail storage, which is rather hacked-in and ideally will be isolated in future. This cache "stage" or type is very similar to `SEQ_CACHE_STORE_RAW` in that it is valid for all strips that share same path to file. Ideally new cache system for such entries can be created which can be compatible with disk cache and cache invalidation mechanisms.

Disk cache design
Disk cache is extension of RAM cache. Storage is configured in user preferences.

Main considerations for disk cache:
 * Images are stored pernamently until invalidated or removed due to disk space limits.
 * Multiple images are stored in 1 file to reduce cold read delays.
 * It must be possible to write images in random order, because rendering is not always sequential.
 * 2 blend files should never use same disk cache path.
 * It should support most features as RAM cache, for example float images. Images must be recovered without any quality loss.

To write images at random order, each cache file has header that maps file contents. This way file can always be appended.

To ensure maximum possible integrity of cache, file name does contain values used by RAM cache to calculate hash, but these values are human readable. Path to cache files consists of blend file name, scene name and RAM cache creation timestamp and strip name. Example of typical path: `untitled.blend_seq_cache/SCScene-1638169261/SQtime.002/8-1920x1080-99%(0)-1.dcf`

To future-proof the design, versioning system is implemented. Each blend file cache directory has `cache_version` file. If version is not matched, all files in cache directory are removed.

Disk cache does support image compression with standard compression algorithms. During testing, these seems to be more effective than image codecs, especially lossless ones.

To avoid unnecessary disk IO, cache directory structure is traversed and `DiskCacheFile` structure representing each file is stored in RAM. Every new cache file that is created is stored in RAM as well. This helps mainly when cache files are invalidated, so whole tree does not have to be traversed possibly during rendering.

Iterator
To make work with sets of strips easier, iterator system can be used. Main design goal is to provide means to inline logic, instead of creating callback functions. `SeqCollection` structure defines a set of strips and can be iterated on by `SEQ_ITERATOR_FOREACH` macro, similar to `LISTBASE_FOREACH` macro. To create `SeqCollection`, most common queries are available, but custom functions can be created. Use of this iterator system is encouraged everywhere where maximum possible performance is not essential. Because `SeqCollection` can be reused multiple times and it can be passed as argument, it may be faster than looping over strips multiple times.

For cases where logic may be complicated such as working on strips not only selected by simple criteria, but also their relationships, there are functions for combining, filtering and expanding `SeqCollection`. This way a bulk of logic responsible for selecting strips that are operated on can be separated from functional logic which does change their data.

Implementation should be well documented, but good example is for example `select_grouped_effect_link`:

/* Select all strips within time range and with lower channel of initial selection. Then select * effect chains of these strips. */ static bool select_grouped_effect_link(SeqCollection *strips,                                       ListBase *seqbase,                                        Sequence *UNUSED(actseq),                                        const int UNUSED(channel)) {  /* Get collection of strips. */  SEQ_filter_selected_strips(strips); const int selected_strip_count = SEQ_collection_len(strips); SEQ_collection_expand(seqbase, strips, query_lower_channel_strips); SEQ_collection_expand(seqbase, strips, SEQ_query_strip_effect_chain); /* Check if other strips will be affected. */  const bool changed = SEQ_collection_len(strips) > selected_strip_count; /* Actual logic. */  Sequence *seq; SEQ_ITERATOR_FOREACH (seq, strips) { seq->flag |= SELECT; }  return changed; }

This function is supplied with collection of strips, contents depend on whether operator is executed from preview where only visible strips are considered, or timeline, in which case all strips will be included. First collection is filtered to only selected strips. Number of strips is saved for later comparison to check if this operation had any effect. Function `SEQ_collection_expand` is used to expand original collection. In practice this means that each strip in sequencer is compared to each strip in provided collection. Callback query function then does selection based on its internal logic. in case of `query_lower_channel_strips`, strip is selected if it is visually under strips from provided collection. in similar fashion, function `SEQ_query_strip_effect_chain` selects every strip, that has effect relationship to strips in provided collection. These are quite complicated selection criteria, that would be hard to parse if done by plain `LISTBASE_FOREACH` iterations. Finally collection is iterated and operator logic is executed - strips are selected.

Note, implementation of this design may change a bit, for example `SEQ_query_strip_effect_chain` may become standalone function, that will take `SeqCollection` as argument. Function `SEQ_collection_expand` is used mainly because it does include logic to prevent infinite loops which are likely in more complicated queries.

Proxy system
To optimize video files playback performance, sequencer uses proxy system which is available in movie clip editor too. Proxies are re-encoded video files with settings to get good scrubbing performance and they can have smaller resolution. Since proxy transcoding is standalone system, its inner workings won't be described in detail here, but rather its integration with sequencer. Same proxy system is used in movie clip editor.

TODO:
 * Image strips
 * Generating editor side, sequencer side
 * How proxies are used

Prefetching
Prefetching can be used to improve consistency of playback speed if image processing demands are unevenly spread. This is done by rendering frames in advance in background job and filling cache up to its limit. This way overhead of drawing code is eliminated too, even though it's usually not significant.

Startup and stop procedure
When prefetching is enabled, function `SEQ_render_give_ibuf` which is responsible for rendering timeline content will try to start prefetching by calling function `seq_prefetch_start`. If prefetch thread is running, but it is suspended, it is resumed. On any update done to sequencer data that affects image output, prefetch must be stopped by calling `SEQ_prefetch_stop`. This is done as a part of cache invalidation, but there are cases when it has to be done explicitly, for example on undo. When image is re-rendered after change, prefetching will restart with new data.

Rendering loop
Because DNA structures may contain runtime data necessary for operation, prefetch operates on `Scene` copy, which contains sequencer data as well. It hosts its own dependency graph to evaluate data for each frame and a copy of `SeqRenderData` rendering context. This context is marked as for prefetching(`is_prefetch_render`), so another prefetching job is not created from prefetching thread. Finally function `seq_prefetch_frames` is executed in a thread. Rendering loop uses dependency loop to evaluate all datablocks. There is call to `BKE_animsys_evaluate_animdata` which I am not entirely sure why it is needed as it shouldn't be. Function `SEQ_render_give_ibuf` is called on copy of scene and rendering context. Rendering loop can be suspended by timeline scrubbing or cache being full. When there are no more frames to render or prefetching is disabled, job is terminated. Loop resumes or starts, when cache is freed again.

Cache handling
Prefetching job has no prior information on where it should stop. This is because each frame may have different size in memory or frames may be missing. Further it does not know about total memory usage since it may change outside of sequencer. Because of this function `seq_prefetch_do_suspend` polls function `seq_prefetch_is_cache_full` on every iteration of rendering loop. If cache is full, thread will be suspended. Once cache is freed through startup procedure signal is sent to prefetching thread to check memory usage and resume render loop if possible. Unlike normal sequencer rendering procedure, prefetch job must store `SEQ_CACHE_STORE_FINAL_OUT` image regardless of memory usage. This is, because function `seq_cache_put_if_possible` would not store image if it's size is greater than available memory and therefore function `seq_prefetch_is_cache_full` would never return `true`. Prefetch job relies on disregarding this limit, but since this will cause thread being suspended, it will never be crossed by large amount.

Another important fact is, that each scene with sequencer data has it's own cache. This means, that calling function `SEQ_render_give_ibuf` with copy of original data would store image with different context and it wouldn't be possible to get it from context that is used to draw UI. This is resolved by referencing original prefetch job in copied scene - `pfjob->scene_eval->ed->prefetch_job = pfjob;` In cache code original data can be accessed with function `seq_prefetch_get_original_context`. This is not great way to resolve the issue though.

Finally, cache freeing strategy is changed from furthest cached frame to "ideally leftmost cached frame outside of prefetching area, otherwise rightmost frame outside of prefetching area". This is, because user can move current frame few frames before, inside or after prefetched area. This can also happen frequently, so freeing strategy should be least disruptive and preserve continuous flow of cached images as much as it can.

To debug prefetching process, cache overlay is useful tool. To continuously update cache overlay, function `SEQ_prefetch_need_redraw` is polled from sequencer drawing code and adds notifier to redraw itself which creates event loop. Event loop is terminated when prefetching is suspended.

Limitations
Currently 3D scene strips can't be prefetched. This is for 2 reasons - They use offscreen rendering, which requires interlock with main frame, which is not implemented. Even if background rendering would be used, problem is that dependency graph used by prefetch job can't be used to create dependency graph for render engine.

Thumbnails
TODO