Page status (reviewing guidelines)
Motion tracking is a new technique available in Blender. It is still under development, and currently supports basic operations for 2D motion tracking, 3D motion tracking, and camera solution. Its already ready to be used in production as validated by "Tears of Steel"
Motion tracking is available in current SVN Trunk and would be included into Blender 2.61 release. It's enabled by default for all platforms and can be used "out-of-box".
Here's brief descriptions of tools of motion tracking currently available in Blender
Supervised 2D tracking
There's no common algorithm which can be used for all kinds of footage, feature points and their motions. Such algorithms can be constructed, but it'll be really slow and it can still fail, so the only way to perform 2D tracking is to manually choose the algorithm of tracking and it's settings. Current defaults would work nice for general footage which isn't very blurry and where feature points aren't getting highly deformed by perspective, i.e.
Improving 2D tracking is already in our TODO list, but it's not high priority at this moment. If you aren't sure about algorithms and settings and don't want to read this document, you can just play with settings and find one which would work for you.
Manual lens calibration using grease pencil and/or grid
All cameras record distorted videos. Nothing can be done about this because of the manner in which optical lenses work. For accurate camera motion, the exact value of the focal length and the "strength" of distortion are needed.
Currently, focal length can be automatically obtained only from the camera's settings or from the EXIF information -- there are no tools inside Blender which can estimate it. But there are some tools which can help to find approximate values to compensate for distortion. There are also fully manual tools where you can use grid which is getting affected by distortion model and deformed cells defines straight lines in the footage. You can also use grease pencil for this - just draw line which should be straight on the footage using poly line brush and adjust distortion values to make grease pencil match lines on the footage.
To calibrate your camera more accurately, use the grid calibration tool from OpenCV. OpenCV is using the same distortion model, so it shouldn't be a problem.
Camera motion solving
Despite that there's no difference in solving camera motion and object motion from math's point of view, only camera solving is currently supported. And it still has some limitations, like unsupported solve of tripod motions or dominant plane motions (where all track-able features belong to one plane). These limitations are planned to be solved in the future.
Basic tools for scene orientation and stabilization
After solve, something is needed to orient real scene in 3D scene for more convenient compositing. There's tools to define floor, scene origin, and X/Y axis to perform scene orientation.
If something is needed to stabilize video from camera to make the final result looks nicer, 2D stabilization is available to help. It stabilizes video from camera which can compensate camera jumps and tilt.
Basic nodes for compositing scene into real footage
Some new nodes were added to compositor to composite scene into footage in easier way. So, there are nodes for 2D stabilization, distortion and undistortion which are easy to use.
Not implemented tools
Some tools aren't available in Blender yet, but they are in our TODO list. So there's currently no support for such things as rolling shutter filtering, object motion solving, motion capturing. But you can try to hack this stuff using currently implemented things.
Movie Clip Editor
Almost all motion tracking tools are concentrated in editor called Movie Clip Editor. Currently it doesn't have any tools which aren't related on motion tracking, but in feature it can be expanded to be used for masking and so, that's why it's called in this more abstract from motion tracking manner.
This editor can be found in list of editor types.
When you'll switch to Movie Clip Editor interface would change in the following way.
Next thing which is logical to do is to open new video clip to start working with. There are several ways to to this:
- Use button from movie editor header
- Use Clip » Open menu
- Use AltO> shortcut
Both of movie files and image sequences can be used in the clip editor. If you're using image sequence there's one limitation on naming of files: last group of numbers should be increasing continuously.
So, when movie clip is loaded into clip editor, extra panels are displayed in the interface.
There are plenty of new tools appeared on the screen and here's short description of all of them.
First of all, it should be mentioned, that camera solver consists of three quite separated steps:
- 2D tracking of footage
- Camera intrinsics (focal length, distortion coefficients) specification/estimation/calibration
- Solving camera, scene orientation, scene reconstruction
Tools in clip editor are split depending on step they're used at, so interface isn't messed up with scene orientation tools when only 2D tracking can be done. Currently displayed tools category can be changed using Mode menu which is in editor header.
But almost all operators can be called from menus, so it's not necessary to change mode every time you want to use a tool which is associated with a different editor mode.
In tracking mode only tools which are related on tracking and camera solving are displayed. Camera solving tools are exposed here because it's after solving you'll most probably want to re-track existing tracks or place new tracks to make solving more accurate.
Tools available in tracking mode
- Add Marker and Move operator places new marker at the position of mouse (which is under the button in this case, not ideal but it's just how things work) and then it can be moved to needed location. When it's moved to needed position, LMB can be used to finish placing new marker. Also, ↵ Enter and Space can be used to finish placing marker.
- But it's faster to use CtrlLMB to place markers directly on the footage. This shortcut will place marker on place you've clicked. One more feature here: until you've released mouse button you can adjust marker position moving mouse and using track preview widget to control how accurate marker is placed.
- Detect Features operator detects all possible to features on current frame and places markers at this features. This operator doesn't take in account other frames, so it can place markers on features which belongs to moving object and if camera is turning away from this shot, no markers would be placed on frames after camera moved away.
- There are several properties for this operator:
- Placement is used to control where to place marker. By default, they'll be added on the whole frame, but you can also outline some interesting for detecting areas with grease pencil and place markers only inside outlined area. That's how "Inside Grease Pencil" placement variant works. You can also outline absolutely not interesting areas (like trees, humans and so) and place markers outside of this areas. That's how "Outside Grease Pencil" placement variant works.
- Margin controls distance from image boundary for created markers. If markers would be placed too close to image boundary, they'll fail to track really soon and they should be deleted manually. To reduce amount of manual cleaning up this parameter can be used.
- Trackability limits minimal trackability for placing markers. This value came from feature detection algorithm and basically it means: low values means most probably this feature would fail to track very soon, high value means it's not much such track. Amount of markers to be added can be controlled with this value.
- Distance defines minimal distance between placed markers. it's needed to prevent markers placed too close to each other (such placement can confuse camera solver).
- Delete Track is quite self-explaining operator which deletes all selected tracks.
- First row of buttons is used to perform tracking of selected tracks. Tracking can happen (in order of buttons):
- Backward one frame
- Backward along the sequence
- Forward along the whole sequence
- Forward one frame
- This operator depends on settings from Tracking Settings panel which would be described later.
- If during sequence tracking algorithm returned failure of tracking some markers, they'll be disabled and tracking will continue for rest of markers. If algorithm returns failure when tracking frame-by-frame, marker wouldn't be disabled and most possible position of feature on new frame would be used.
- Clear After deletes all tracked and keyframed markers after current frame for all selected tracks.
- Clear Before deletes all tracked and keyframed markers before current frame for all selected tracks.
- Clear clears all markers except current from all selected tracks.
- Join operator joins all selected tracks into one. Selected tracks shouldn't have common tracked or keyframed markers at the same frame.
Camera Motion operator solves motion of camera using all tracks placed on the footage and two keyframes specified on this panel. There are some requirements:
- There should be at least 8 common tracks on the both of keyframes
- It should be well noticeably parallax effect between this two keyframes
If everything came smooth during solve, average reprojection error would be reported to the information space and to clip editor header. Reprojeciton error means average distance between reconstructed 3D position of tracks projected back to footage and original position of tracks. Basically, reprojection error below 0.3 means accurate reprojection, 0.3-3.0 means quite nice solving which still can be used. Values above 3 means some tracks should be tracked more accurate or that values for focal length or distortion coefficients were set incorrectly.
Refine option specifies which parameters should be refined during solve. Such kind of refining is useful when you aren't sure about some camera intrinsics and solver would try to find best parameter for that intrinsics. But you still have to know approximate initial values - it'll fail to find correct values if they were set completely incorrect initially.
This panel contains single operator and it's settings. This operator cleans up bad tracks: tracks which aren't tracked long enough or which failed to reconstruct accurate. Threshold values can be specified from slides below the button. Also, several actions can be performed for bad tracks:
- They can be simply selected
- Bad segments of tracked sequence can be removed
- The whole tracks can be deleted
This panel currently contains the single operatorwhich sets currently editing clip as camera background for all visible 3D viewports. If there's no visible 3D viewports or clip editor is opened in full screen, nothing will happen.
Properties available in tracking mode
Grease Pencil Panel
It's a standard grease pencil panel where new grease pencil layers and frames can be controlled. There's one difference in behavior of grease pencil from other areas - when new layer is created "by-demand" (when making stroke without adding layer before this) default color for layer would be set to pink. It makes stroke easy to notice on all kinds of movies.
This panel contains list of all objects which can be used for tracking, camera or object solving. By default there's only one object in this list which is used for camera solving. It can't be deleted and other objects can't be used for camera solving, all added objects are used for object tracking and solving only. These objects can be referenced from Follow Track and Object Solver constraints. Follow Track uses camera object by default.
New objects can be added usingand active object can be deleted by button. Text entry at the bottom of this panel is used to rename active object.
If some tracks were added and tracked to wrong object, they can be copied to another object using Track » Copy Tracks and Track » Paste Tracks.
Usage for all kind of objects (used for camera and object tracking) is the same: track features, set camera data, solve motion. Camera data is sharing between all objects and refining of camera intrinsics happens when solving camera motion only.
First of all, track name can be changed in this panel. Track names are used for linking tracking data to other areas like follow track constraint.
Next thing which can be controlled here is marker's enabled flag (using button with eye icon). If marker is disabled, it's position wouldn't be used neither by solver nor by constraints.
Button with lock at the right of button with eye means track is locked. Locked tracks means it can't be edited at all. This helps to prevent accidental changes to tracks which are "finished" (tracked accurate along the whole footage).
Next widget placed on this panel is called "Track Preview" and it displays content of pattern area. This helps to check how good tracking is happening (control there's no slides from original position) and also helps to move track back to needed position. Moving of track can happen directly from this widget by mouse slide.
If anchor is used (position on image which is tracking is different from position which is used for parenting) preview widget will display area around anchor position. Such configuration helps for masking some things when there's no good feature at position where mask corner should be placed. Details of this technique would be written later.
There's small area below the preview widget which can be used to enlarge vertical size of preview widget (area is highlighted by two horizontal lines).
Next setting is channels control. Tracking happens in grayscale space, so a high contrast between feature and background yields more accurate tracking. In such cases disabling some color channels can help.
The last thing is custom color and preset for it. This setting overrides default marker color used in clip editor and 3D viewport and it helps to distinguish different type of features (for example features on far plane or near plane and so). Color is also can be used for "grouping" tracks so the whole group of tracks can be selected by color using Select Grouped operator.
- Marker Tip 1: To use good points for tracking, use points in the middle of the footage timeline and track backwards and forwards from there. This will provide greater chance of the marker and point staying in the camera shot.
Camera Data Panel
This panel contains all settings of camera used for screening movie which is currently editing in clip editor.
First of all, predefined settings can be used here. New presets can be added or unused presets can be deleted. But such settings as distortion coefficients and principal point aren't included into presets and should be filled in even if camera presets were used.
- Focal Length is already self-described, it's a focal length with which movie was shoot. It can be set in millimeters and pixels. In most cases focal length is given in millimeters, but sometimes (for example in some tutorials in the Internet) it's given in pixels. In such cases it's possible to set it directly in units it's known in.
- Sensor Width is a width of CCD sensor in camera. This value can be found in camera specifications.
- Pixel Aspect Ratio is a pixel aspect of CCD sensor. This value can also be known from camera specifications, but can also be guessed. For example, you know that footage should be 1920x1080 but images itself are 1280x1080. In this case pixel aspect is:
1920 / 1280 = 1.5
- Optical Center is an optical center of lens used in camera. In most cases it's equals to image center, but it can be differ in some special cases. Check camera/lens specifications in such cases. To set optical center to center of image, there's button below sliders.
- Undistortion K1, K2 and K3 is a coefficients used to compensate lens distortion happened on movie shooting. Currently this values can be tweaked by hand only (there's no calibration tools yet) using tools available in Distortion mode. Basically, just tweak K1 until solving would be most accurate for known focal length (but also take grid and grease pencil into account to prevent "impossible" distortion)
This panel contains all settings which controls things displayed in clip editor.
- R, G, B and B/W buttons at the top of this panel are used to control color channels used for frame preview and to make the whole frame grayscale. It's needed because tracking algorithm works with grayscale image and it's not always obvious to see which channels disabled will increase contrast of feature point and reduce noise.
- Pattern can be used to disable displaying of rectangles which are correspond to pattern areas of tracks. In some cases it helps to make clip view more clear and check how good tracking is.
- Search can be used to disable displaying of rectangles which are correspond to search areas of tracks. In some cases it helps to make clip view more clear and check how good tracking is. Search areas for selected tracks only would be displayed.
- Pyramid makes the highest pyramid level be visible. What pyramid itself is would be described later in Tracking Settings panel section, but basically it helps to determine how much track is allowed to move from one frame to another.
- Track Path and Length controls displaying of path of tracks. So way in which tracks are moving can be visible looking at the only one frame. It helps to determine if track jumps from it's position or not.
- Disabled Tracks makes possible to hide all tracks which are disabled on current frame. This helps to make view more clear to control if tracking happens accurate enough.
- Bundles makes sense after solving movie clip and it works in the following way: solved position of each track gets projected back to movie clip and displayed as small point. Color of point depends on distance of projected coordinate and original coordinate: if they are close enough point would be green, otherwise it'll be red. This helps to find tracks which weren't solved nicely and need to be tweaked.
- Track Names and Status displays such information as track name and status of track (if it's keyframed, disabled, tracked or estimated). Names and status for selected tracks is displayed.
- Compact Markers. Way in which markers are displayed (black outline and yellow foreground color) makes tracks be visible on all kind of footages (both dark and light). But sometimes it can be annoying and this option will make displaying of marker in more compact manner - outline would be replaced by dashed black lines drawing on top of foreground, so markers areas would be 1px thick only.
- Grease pencil means if grease pencil strokes are allowed to be displayed and made.
- Mute changes displaying on movie frame itself with black square, It helps to find tracks which are tracked inaccurate or which weren't tracked at all.
- Grid (available in distortion mode only) displays grid which is originally orthographic but was affected by distortion model. This grid can be used for manual calibration - distorted lines of grids are equal to straight lines in the footage.
- Manual Calibration (available in distortion mode only) applies distortion model for grease pencil strokes. This option also helps to perform manual calibration. More detailed description of this process would be made later.
- Stable (available in reconstruction mode only). This option makes displaying frame be affected by 2D stabilization settings. It's only preview option which doesn't actually changes footage itself.
- Lock to Selection makes editor be displaying selected tracks on the same screen position along the whole footage during playback or tracking. This option helps to control tracking process and stop it when track is starting sliding off or when it jumped.
- Display Aspect Ratio changes aspect ratio for displaying only. It does not affect on tracking or solving process.
Tracking Settings Panel
This panel contains all settings for 2D tracking algorithms. Depending on which algorithm is used, different settings would be displayed, but there are also few common for all trackers settings:
Adjust Frames controls which patterns are getting tracking, if be more precise, pattern from which frame is getting tracked. Here's an example which should make things more clear.
Tracker algorithm receives two images inside search area and position of point to be tracked in first image. And tracker tries to find position of point from first image on second image.
Now, how tracking of sequence happens. Second image is always image created from frame at which position of marker isn't known (next tracking frame). But different first image can be send to tracker. Most commonly used combinations:
- Image created from frame on which track was keyframed. This configuration prevents sliding from original position (because position which corresponds most to the original pattern is returning by tracker), but it can lead to small jumps and can lead to failures when feature point is getting deformed due to camera motion (perspective transformation, for example). Such configuration is used if Adjust Frames is set to 0.
- Image created from current frame is sending as first image to the tracker. In this configuration pattern is tracking between two neighbor frames and it allows to deal with cases of high transformations of feature point but can lead to sliding from original position, so it should be controlled. Such configuration is used if Adjust Frames is set to 1.
If Adjust Frames is greater than 1, behavior of tracker would be like keyframes for tracks are creating every Adjust Frames frames and tracking between keyframed image and next image is used.
Speed can be used to control speed of sequence tracking. This option makes nothing with quality of tracking it's just helps to control if tracking happens accurate. In most of cases tracking happens much faster than realtime playing and it's difficult to notice when track began to slide off from position. In such cases Speed can be set to Double or Half to add some delay between tracking two frames, so slide off would be noticed earlier and tracking process can be cancelled to adjust positions of tracks.
Frames Limit controls how many frames can be tracked when Track Sequence operator is called. So, each Track Sequence operation would track maximum Frames Limit frames. This also helps to notice slide off of tracks and correct them.
Margin can be used to make tracks disabled when they are becoming too close to image boundary. This slider controls "too close" in pixels.
KLT tracker options
KLT tracker is an algorithm used by default. It allows to track most kinds of feature points and it's motion. It's using pyramid tracking which works in the following way. Algorithm tracks larger image than pattern first to find general direction of motion. Then it tracks a bit smaller image to refine position from first step and make final position more accurate. This continues several times. Number of steps of such tracking is equal to Pyramid Level option and we tell that on first step tracking happens for highest pyramid level. So Pyramid Level=1 is equal to pattern itself, and each next level doubles tracking image by 2.
Search area should be larger than highest pyramid level and "free space" between search area and highest pyramid level defines how much feature can be moved from one frame to another.
Default settings should work in most of general cases, but sometimes pyramid level should be changed. For example, when footage is blurry, adding extra pyramid levels helps to track them.
This algorithm can fail in situations when feature point is moving in one direction and texture around feature point is moving in another direction.
SAD tracker options
On each step SAD tracker reviews the whole search area and finds pattern on second image which is mostly like pattern which is getting tracking. This works pretty fast, but can fail in several cases. For example, when there're another feature point which looks like tracking feature point appears in search area. In this case SAD will lead to jump of track from one feature to another.
Correlation defines threshold value for correlation between two patters which is still considering as successful tracking. 0 means there's no correlation at all, 1 means correlation is full.
There's one limitation: currently, it works for features 16x16 only.
This panel contains numerical settings for marker position, pattern and search areas dimensions, offset of anchor point from pattern center. All sliders are self-explainable.
Proxy / Timecode Panel
This panel contains options used for image proxies and timecodes for movies.
Proxy allows to display images with lower resolution in clip editor. This can be helpful in cases when tracking of 4K footage is happening on machine with small amount of RAM.
First four options are used to define which resolutions of proxy images should be built. Currently it's possible to build images 25%, 50%, 75% and 100% from original image size. Proxy size of 100% can be used for movies which contains broken frames which can't be decoded.
Build Undistorted means that proxy builder would also create images from undistorted original images for sizes set above. This helps to have faster playback of undistorted footage.
Generated proxy images are encoding using JPEG and quality of JPEG codec is controlled by Quality slider.
By default, all generated proxy images are storing to <path of original footage>/BL_proxy/<clip name> folder, but this location can be set by hand using Proxy Custom Directory option.
will regenerate proxy images for all sizes set above and regenerate all timecodes which can be used later.
Use Timecode Index can (and better be used) for movie files. Basically, timecode makes frame search faster and more accurate. Depending on your camera and codec different timecodes can make better result.
Proxy Render Size defines which proxy image resolution is used for display. If Render Undistorted is set, then images created from undistorted frame would be used. If there's no generated proxies, render size is set to "No proxy, full render" and render undistorted is enabled, undistortion will happen automatically on frame draw.
Tools available in reconstruction mode
There's one extra panel which is available in reconstruction mode - 2D stabilization panel.
This panel is used to define data used for 2D stabilization of shot. Several options are available in this panel.
First of all, it's list of tracks used to compensate camera jumps, or location. It works in the following way: it gets tracks from list of tracks used for location stabilization and finds median point of all these tracks on first frame. On each other frame algorithm makes this point have the same position in screen coordinates by moving the whole frame. In some cases it's not needed to fully compensate camera jumps and Location Influence can be used in such cases.
Camera can also rotate a bit adding some tilt to the footage. There's Stabilize Rotation option to compensate this tilt. Extra single track is needed to set for this and it works in the following way. On first frame of movie, this track is getting connected with median point of tracks from list above and angle between horizon and this segment is keeping constant along the whole footage. Amount of rotation applying on the footage can be controlled by Rotation Influence.
If camera jumps a lot, it'll be noticeable black areas near image boundaries. To get rid of these black holes, there's Autoscale option which finds smallest scale, applying which on the footage all black holes near image boundaries would eliminate. There's option to control maximal scale factor (Maximal Scale) and amount of scale applying on the footage (Scale Influence)