From BlenderWiki

Camera tracking integration: Documentation

Thesaurus

Based on article from Keir Mierle.

Marker

A 2D element that is driven by a tracking algorithm. Defines an image point (target) on a frame. Tracking attempts to match a pattern box around that point at all subsequent (or precedent) frames, within a given search area of its position at each frame.

Properties:

• Is rendered on an overlay of the viewport, hence it can never be occluded (but may be hidden by setting an attribute).
• May be driven by a tracking algorithm.
• Identifies one point on each frame.
• Has an associated “pattern” box, symmetrical and (image) axis aligned with respect to the point, which defines the portion of image that is matched from frame to frame when tracking.
• Has an associated axid-aligned “search” box, which restricts the portion of image at the where the pattern may be matched when tracking
• UI actions:
  • Create with default attributes under mouse with one click.
  • Drag point (pattern and search boxes follow rigidly).
  • Resize pattern.
  • Drag, resize search box leaving point and search box unchanged.
  • Hide as a whole; hide/show boxes.

Attributes (type):

• x, y, [z] (float). 3D coordinates (normally normalized image coordinates to discount the image resolution). Z, if present, is normally the normalized depth of the image plane (can be used to define a scale). Can be discounted altogether if the scene graph allows for pure 2D point-like elements.
• frame_num (integer). Number of of frame this marker is associated with. Marker could belong to only one frame and can't change it.

Behavior

Manually-set position always overrides position driven by the tracker algorithm (this may be “free” depending on how the scene graph implements driving and overrides). This is necessary to allow for manually correcting bad automatic tracks.

Track

Set of markers which appears after tracking one the single marker.Track defines movement of target along the time.

Properties

• Is a fully-qualified, rendered and animated member of the scene graph. E.g it can be parented under any transform nodes (in normal use it is parented under a camera).
• In Blender-side terms track could be associated with bundle after reconstructing, This happens automatically by reconstruction algorithm and user can't affect on this property.

Attributes (type):

• pattern_width, pattern_height, pattern_offset_x, pattern_offset_y (float): The width, height and 2D offsets of the pattern area. These are normally specified in pixels. Global setting for all markers in this track.
• search_width, search_height, search_offset_x, search_offset_y (float): Width, height and 2D offsets in pixels of the search box with respect to the point. Global setting for all markers in this track.
• markers (set): set of all markers belongs to this track.

This entity is "faked" in libmv (marekrs have got track_id which specifies markers from the same track), but it's real entity in Blender side.

Bundles

Bundles are 3D point-like elements that represent the estimated location in space associated to a set of 2D points. Examples:

• The triangulated 3D location of 2 or more views of a physical points in 2 or more cameras at the same time.
• The triangulated 3D location of multiple (perhaps tracked) 2D views of the same physical point at different times in one camera.

Properties

• Each bundle is normally associated with at least one marker (but no marker can be associated with more than one bundle, this is a constrain that should be enforced by the scene graph). The form of this association depends on the scene graph implementation (an operator or a predicate). At a minimum it needs to be bi-directional: a bundle must “know” its markers and viceversa, as without both direct and reverse indexing many important photogrammetric operations in the scene graph have quadratic time complexity.
• Bundles are rendered and animated shapes in the scene graph, with their animation possibly set by a SFM/Bundler algorithm and/or point constrain objects (see below). Unlike markers, they are rendered in a 3D view, e.g. as small spheres. As part of the scene graph, they behave like any other 3D shape as far as coordinate transformations are concerned. It normally useful to distinguish (via an attribute of the object) between “solved” bundles, whose location has been set by an algorithm, and “unsolved” ones.

Visualization

Tracking entity relation schema

Implementation

DNA structures

Data structure for marker

typedef struct MovieTrackingMarker {
	float pos[2];	/* 2d position of marker on frame (in unified 0..1 space) */
	int framenr;	/* number of frame marker is associated with */
	int flag;		/* Marker's flag (alive, ...) */
} MovieTrackingMarker;

Data structure for track

typedef struct MovieTrackingTrack {
	struct MovieTrackingTrack *next, *prev;
 
	char name[24];
 
	/* ** setings ** */
	float pat_min[2], pat_max[2];		/* positions of left-bottom and right-top corners of pattern (in unified 0..1 space) */
	float search_min[2], search_max[2];	/* positions of left-bottom and right-top corners of search area (in unified 0..1 space) */
	float offset[2];					/* offset to "parenting" point */
 
	int pad;
 
	/* ** track ** */
	int markersnr;					/* count of markers in track */
	int last_marker;				/* most recently used marker */
	int pad2;
	MovieTrackingMarker *markers;	/* markers in track */
 
	/* ** bundle ** */
	float bundle_pos[3];
 
	/* ** UI editing ** */
	int flag, pat_flag, search_flag;	/* flags (selection, ...) */
	short transflag;					/* transform flags */
	char pad3[2];
	float color[3];						/* custom color for track */
} MovieTrackingTrack;

Modules

One new module and some new files were created during this GSoC project which I'd like to describe a bit.

Module: space_clip

Location: source/blender/editor/space_clip/

Description: implementation of new editor space in which artists are interacting with movie clips,

It it implementation of new space called "Movie Clip Editor". It's not supposed to be tracking-related only, so probably name isn't fit it's destination yet, but there are several plans of new features so it's name would be totally fine.

This module is split into several files:

• clip_buttons.c - this file contains implementation of some UI widgets and panels. For example "Marker" panel and "Track preview" widget Py-API binding are situated here.
• clip_draw.c - contains all functions needed for drawing all things in main area of editor.
• clip_editor.c - some general utility functions which can be used all over. It's mostly small "abstraction" between space API and kernel which helps to access image buffers, zoom, dimension and so from SpaceClip-related operators to avoid direct accessing some fields in this structure which can be unsafe.
• clip_header.c - contains functions to toggle Tools/Properties regions.
• clip_intern.h - header file for functions which are for internal use but which are shared between some files in SpaceClip. For example this file contains declarations of operators.
• clip_ops.c - contains implementation of generic clip-related functions like panning, zoom and so. All that operators which aren't match-moving related.
• space_clip.c - declaration file of SpaceClip.
• tracking_ops.c - all match-moving related operators are implemented here.

Module: libmv

Location: extern/libmv

Description: bundled version of libmv and C-API for it.

Nothing really special, just minimal subset of libmv from it's original repo is stored here. Also there's implementation of C-API interface. Also simple script to update bundled sources is here.

File: tracking.c

Location: sources/blender/blenkernel/intern/tracking.c

Description: implementation of "core" for tracking.

This file contains all match-moving related stuff on which operators for SpaceClip are based.

File: movieclip.c

Location: sources/blender/blenkernel/intern/movieclip.c

Description: file with movie-clip related functions.

Includes loading movie clips (from both of movies and sequences types), caching mechanisms are used here. Also contains such helpers functions as selection control, scopes and so.

File: moviecache.c

Location: sources/blender/blenkernel/moviecache.c

Description: cache for ibuf-related areas as SpaceClip

This file was based on seqcache.c from Peter Schlaile, but it's supposed to be a bit more "abstract". For example seqcache requires Strip as key for cache which isn't so useful for case of SpaceClip. All this area-related stuff is supposed to be implemented as small callback functions for GHash, so i.e. MovieClip has it's own small ghash-key function and seqcache would have small key function, but all other stuff like memory limiter and so would be the same for this two areas.