From BlenderWiki

Jump to: navigation, search
Blender3D FreeTip.gif
IMPORTANT! Do not update this page!
We have moved the Blender User Manual to a new location. Please do not update this page, as it will be locked soon.

Motion Tracking (sledování pohybu kamery)

Úvod

Motion tracking je novou technikou dostupnou v Blenderu. Stále je ve vývoji a zatím podporuje jen základní operace pro sledování pohybu kamery. Nicméně již je připraven pro použití.

Začínáme

Motion tracking je dostupný v aktuálním SVN trunku a bude zahrnut v Blendru verze 2.61. Je dostupný pro všechny platformy a může být použit "out-of-box".

Zde je stručný popis již v Blenderu dostupných nástrojů pro motion tracking.

Dohlížení na 2D tracking

Neexistuje žádný obecný algoritmus, který by mohl být použit pro všechny možné záběry, charakteristické body a jejich pohyb. Takový algoritmus by mohl být vytvořen, nicméně by byl velmi pomalý a stejně by nemusel fungovat bezchybně. Jedinou možností jak provádět 2D tracking je, manuálně volit vhodný algoritmus a jeho nastavení. Současné výchozí nastavení by mělo pěkně fungovat pro většinu záběrů, které nejsou moc rozmazány a kde charakteristické body příliž nedeformuje perspektivní zobrazení, atd.

Vylepšený 2D tracking již je v TODO seznamu, ale aktuálně nepatří mezi priority. Pokud si nejste jisti algoritmy, jejich nastavením a nechcete číst tento dokument, můžete si pohrát s nastavením a najít tak to, které vám přinese požadovaný výsledek.

Manuální kalibrace objektivu pomocí grease pencil a/nebo mřížky

Všechny kamery, vlivem chyb objektivů, zaznamenávají zkreslený obraz, s čímž nic udělat nemůžeme. Pro zjištění přesného pohybu kamery potřebujeme znát přesnou hodnotu ohniskové vzdálenosti a "sílu" zkreslení.

V současnosti můžeme ohniskovou vzdálenost zjistit z nastavení kamery nebo z EXIFu -- Blender si tuto hodnotu neumí zjistit sám. Je tu však několik nástrojů, které mohou pomoci najít přibližné hodnoty a kompenzovat zkreslení. K dispozici je také plně manuální nástroj, který umožňuje použití sítě deformované modelem zkreslení, která definuje přímky v záběru. Můžete také použít grease pencil - stačí nakreslit linku pomocí "poly line brush", která by měla být přímo na záběru, a upravit hodnoty zkreslení tak, aby korespondovala s linií v záznamu.

Jedinou možností pro přesnější kalibraci vaší kamery je použití nástroje "grid calibration" z knihovny OpenCV. OpenCV používá stejný model zkreslení, takže by to neměl být problém.

Řešení pohybu kamery

Přesto že není žádný rozdíl v detekci pohybu kamery a pohybu objektů, alespoň z matematického úhlu pohledu, je zatím k dispozici jen pohyb kamery. Toto řešení má zatím své hranice, mezi které patří chybějící podpora pohybu kamery na stativu nebo pohyb dominantní roviny (kde je všechen pohyb vstažen k jedné rovině). V budoucnu by tyto nedostatky měly být odstraněny.

Základní nástroje pro orientaci a stabilizaci scény

Po detekci pohybu je potřeba nastavit správnou orientaci reálné scény v 3D prostoru, kvůli pohodlnější práci. K dispozici je nástroj pro definici "podlahy", středu scény a x/y-ové osy, které umožňují scénu orientovat.

Někdy je také potřeba videozáznam z kamery stabilizovat, aby finální výsledek vypadal lépe. K dispozici je 2D stabilizace, která může kompenzovat skoky a naklonění kamery.

Základní uzly pro zakomponování scény do reálného záznamu

Pro jednodušší komponování 3D scény do videozáznamu, byly do kompozitoru přidány některé nové uzly. Jsou zde uzly pro 2D stabilizaci, deformaci a reformaci, které je jednoduché použít.

Neinplementované nástroje

Některé nástroje zatím nejsou v Blenderu dostupné, ale jsou v TODO seznamu. Zatím nejsou podporovány takové věci, jako filtr pohybového zkreslení (rolling shutter filter), řešení pohybu objektu, nebo motion capturing. Nicméně se můžete tyto věci nahradit již implementovanými funkcemi.

Manuál

Movie Clip Editor

Almost all motion tracking tools are concentrated in editor called Movie Clip Editor. Currently it doesn't have any tools which aren't related on motion tracking, but in feature it can be expanded to be used for masking and so, that's why it's called in this more abstract from motion tracking manner.

This editor can be found in list of editor types.

Editor type menu

When you'll switch to Movie Clip Editor interface would change in the following way.

Movie Clip Editor interface

Next thing which is logical to do is to open new video clip to start working with. There are several ways to to this:

  • Use Open button from movie editor header
  • Use Clip » Open menu
  • Use AltO> shortcut

Both of movie files and image sequences can be used in the clip editor. If you're using image sequence there's one limitation on naming of files: last group of numbers should be increasing continuously.

So, when movie clip is loaded into clip editor, extra panels are displayed in the interface.

Movie Clip Editor with opened clip

There are plenty of new tools appeared on the screen and here's short description of all of them.

First of all, it should be mentioned, that camera solver consists of three quite separated steps:

  • 2D tracking of footage
  • Camera intrinsics (focal length, distortion coefficients) specification/estimation/calibration
  • Solving camera, scene orientation, scene reconstruction

Tools in clip editor are split depending on step they're used at, so interface isn't messed up with scene orientation tools when only 2D tracking can be done. Currently displayed tools category can be changed using Mode menu which is in editor header.

Movie Clip Editor mode menu

But almost all operators can be called from menus, so it's not necessary to change mode every time you want to sue tool which is displayed in panels on differ editor mode.

In tracking mode only tools which are related on tracking and camera solving are displayed. Camera solving tools are exposed here because it's after solving you'll most probably want to re-track existing tracks or place new tracks to make solving more accurate.

Tools available in tracking mode

Markers panel
  • Add Marker and Move operator places new marker at the position of mouse (which is under the button in this case, not ideal but it's just how things work) and then it can be moved to needed location. When it's moved to needed position, LMB Template-LMB.png can be used to finish placing new marker. Also, ↵ Enter and Space can be used to finish placing marker.
But it's faster to use CtrlLMB Template-LMB.png to place markers directly on the footage. This shortcut will place marker on place you've clicked. One more feature here: until you've released mouse button you can adjust marker position moving mouse and using track preview widget to control how accurate marker is placed.
  • Detect Features operator detects all possible to features on current frame and places markers at this features. This operator doesn't take in account other frames, so it can place markers on features which belongs to moving object and if camera is turning away from this shot, no markers would be paced on frames after camera moved away.
There are several properties for this operator:
Placement is used to control where to place marker. By default, they'll be added on the whole frame, but you can also outline some interesting for detecting areas with grease pencil and place markers only inside outlined area. That's how "Inside Grease Pencil" placement variant works. You can also outline absolutely not interesting areas (like trees, humans and so) and place markers outside of this areas. That's how "Outside Grease Pencil" placement variant works.
Margin controls distance from image boundary for created markers. If markers would be placed too close to image boundary, they'll fail to track really soon and they should be deleted manually. To reduce amount of manual cleaning up this parameter can be used.
Trackness limits minimal trackness for placing markers. This value came from feature detection algorithm and basicly it means: low values means most probably this feature would fail to track very soon, high value means it's not much such track. Amount of markers to be added can be controlled with this value.
Distance defines minimal distance between placed markers. it's needed to prevent markers placed too close to each other (such placement can confuse camera solver).
  • Delete Track is quite self-explaining operator which deletes all selected tracks.
Track panel
  • First row of buttons is used to perform tracking of selected tracks. Tracking can happen (in order of buttons):
    • Backward one frame
    • Backward along the sequence
    • Forward along the whole sequence
    • Forward one frame
This operators depends on settings from Tracking Settings panel which would be described later.
If during sequence tracking algorithm returned failure of tracking some markers, they'll be disabled and tracking will continue for rest of markers. If algorithm returns failure when tracking frame-by-frame, marker wouldn't be disabled and most possible position of feature on new frame would be sued.
  • Clear After deletes all tracked and keyframed markers after current frame for all selected tracks.
  • Clear Before deletes all tracked and keyframed markers before current frame for all selected tracks.
  • Clear clears all markers except current form all selected tracks.
  • Join operator joins all selected tracks into one. Selected tracks shouldn't have common tracked or keyframed markers at the same frame.
Solve panel

Camera Motion operator solves motion of camera using all tracks placed on the footage and two keyframes specified on this panel. There are some requirements:

  • There should be at least 8 common tracks on the both of keyframes
  • It should be well noticeably parallax effect between this two keyframes

If everything came smooth during solve, average reprojection error would be reported to the information space and to clip editor header. Reprojeciton error means average distance between reconstructed 3D position of tracks projected back to footage and original position of tracks. Basically, reprojection error below 0.3 means accurate reprojection, 0.3-3.0 means quite nice solving which still can be used. Values above 3 means some tracks should be tracked more accurate or that incorrect values for focal length or distortion coefficients were set incorrectly.

Refine option specifies which parameters should be refined during solve. Such kind of refining is useful when you aren't sure about some camera intrinsics and solver would try to find best parameter for that intrinsics. But you still have to know approximate initial values - it'll fail to find correct values if they were set completely incorrect initially.

Cleanup Panel

This panel contains single operator and it's settings. This operator cleans up bad tracks: tracks which aren't tracked long enough or which failed to reconstruct accurate. Threshold values can be specified from slides below the button. Also, several actions can be performed for bad tracks:

  • They can be simply selected
  • Bad segments of tracked sequence can be removed
  • The whole tracks can be deleted
Clip Panel

This panel currently contains the single operator Set as background which sets currently editing clip as camera background for all visible 3D viewports. If there's no visible 3D viewports or clip editor is opened in full screen, nothing will happen.

Properties available in tracking mode

Grease Pencil Panel

It's a standard grease pencil panel where new grease pencil layers and frames can be controlled. There's one difference in behavior of grease pencil from other areas - when new layer is created "by-demand" (when making stroke without adding layer before this) default color for layer would be set to pink. It makes stroke easy to notice on all kinds of movies.

Track Panel
Track Panel in clip editor

First of all, track name can be changed in this panel. Track names are used for linking tracking data to other areas like follow track constraint.

Next thing which can be controlled here is marker's enabled flag (using button with eye icon). If marker is disabled, it's position wouldn't be used neither by solver nor by constraints.

Button with lock at the right of button with eye means track is locked. Locked tracks means it can't be edited at all. This helps to prevent accidental changes to tracks which are "finished" (tracked accurate along the whole footage).

Next widget placed on this panel is called "Track Preview" and it displays content of pattern area. This helps to check how good tracking is happening (control there's no slides from original position) and also helps to move track back to needed position. Moving of track can happen directly from this widget by mouse slide.

If anchor is used (position on image which is tracking is differ from position which is used for parenting) preview widget will display area around anchor position. Such configuration helps for masking some things when there's no good feature at position where mask corner should be placed. Details of this technique would be written later.

There's small area below the preview widget which can be used to enlarge vertical size of preview widget (area is highlighted by two horizontal lines).

Next setting is channels control. Tracking happens in grayscale space, so it can be not enough contrast between feature and background to perform accurate tracking. In such cases disabling some color channels can help.

The last thing is custom color and preset for it. This settings overrides default marker color used in clip editor and 3D viewport and it helps to distinguish different type of features (for example features on far plane or near plane and so). Color is also can be used for "grouping" tracks so the whole group of tracks can be selected by color using Select Grouped operator.

Camera Data Panel

This panel contains all settings of camera used for screening movie which is currently editing in clip editor.

First of all, predefined settings can be used here. New presets can be added or unused presets can be deleted. But such settings as distortion coefficients and principal point aren't included into presets and should be filled in even if camera presets were used.

  • Focal Length is already self-described, it's a focal length with which movie was shoot. It can be set in millimeters and pixels. In most cases focal length is given in millimeters, but sometimes (for example in some tutorials in the Internet) it's given in pixels. In such cases it's possible to set it directly in units it's known in.
  • Sensor Width is a width of CCD sensor in camera. This value can be found in camera specifications.
  • Pixel Aspect Ratio is a pixel aspect of CCD sensor. This value can also be known from camera specifications, but can also be guessed. For example, you know that footage should be 1920x1080 but images itself are 10280x1080. In this case pixel aspect is:
1920 / 1280 = 1.5
  • Optical Center is an optical center of lens used in camera. In most cases it's equals to image center, but it can be differ in some special cases. Check camera/lens specifications in such cases. To set optical center to center of image, there's Center button below sliders.
  • Undistortion K1, K2 and K3 is a coefficients used to compensate lens distortion happened on movie shooting. Currently this values can be tweaked by hand only (there's no calibration tools yet) using tools available in Distortion mode. Basically, just tweak K1 until solving would be most accurate for known focal length (but also take grid and grease pencil into account to prevent "impossible" distortion)
Display Panel

This panel contains all settings which controls things displayed in clip editor.

  • Pattern can be used to disable displaying of rectangles which are correspond to pattern areas of tracks. In some cases it helps to make clip view more clear and check how good tracking is.
  • Search can be used to disable displaying of rectangles which are correspond to search areas of tracks. In some cases it helps to make clip view more clear and check how good tracking is. Search areas for selected tracks only would be displayed.
  • Pyramid makes the highest pyramid level be visible. What pyramid itself is would be described later in Tracking Settings panel section, but basically it helps to determine how much track is allowed to move from one frame to another.
  • Track Path and Length controls displaying of path of tracks. So way in which track s are moving can be visible looking at the only one frame. It helps to determine if track jumps from it's position or not.
  • Disabled Tracks makes possible to hide all tracks which are disabled on current frame. This helps to make view more clear to control if tracking happens accurate enough.
  • Bundles makes sense after solving movie clip and it works in the following way: solved position of each track gets projected back to movie clip and displayed as small point. Color of point depends on distance of projected coordinate and original coordinate: if they are close enough point would be green, otherwise it'll be red. This helps to find tracks which weren't solved nicely and need to be tweaked.
  • Track Names ans Status displays such information as track name and status of track (if it's keyframed, disabled, tracked or estimated). Names and status for selected tracks is displayed.
  • Compact Markers. Way in which markers are displayed (black outline and yellow foreground color) makes tracks be visible on all kind of footages (both dark and light). But sometimes it can be annoying and this option will make displaying of marker in more compact manner - outline would be replaced by dashed black lines drawing on top of foreground, so markers areas would be 1px thick only.
  • Grease pencil means if grease pencil strokes are allowed to be displayed and made.
  • Mute changes displaying on movie frame itself with black square, It helps to find tracks which are tracked inaccurate or which weren't tracked at all.
  • Grid (available in distortion mode only) displays grid which is originally orthographic but was affected by distortion model. This grid can be used for manual calibration - distorted lines of grids are equal to straight lines in the footage.
  • Manual Calibration (available in distortion mode only) applies distortion model for grease pencil strokes. This option also helps to perform manual calibration. More detailed description of this process would be made later.
  • Stable (available in reconstruction mode only). This options makes displaying frame be affected by 2D stabilization settings. It's only preview option which doesn't actually changes footage itself.
  • Lock to Selection makes editor be displaying selected tracks on the same screen position along the whole footage during playback or tracking. This option helps to control tracking process and stop it when track is starting sliding off or when it jumped.
  • Display Aspect Ratio changes aspect ratio for displaying only. It does not affect on tracking or solving process.
Tracking Settings Panel
Common options

This panel contains all settings for 2D tracking algorithms. Depending on which algorithm is used, different settings would be displayed, but there are also few common for all trackers settings:

Adjust Frames controls which patterns are getting tracking, if be more precious, pattern from which frame is getting tracked. Here's an example which should make things more clear.

Tracker algorithm receives two images inside search area and position of point to be tracked in first image. And tracker tries to find position of point from first image on second image.

Now, how tracking of sequence happens. Second image is always image created from frame at which position of marker isn't known (next tracking frame). But different first image can be send to tracker. Most commonly used combinations:

  • Image created from frame on which track was keyframed. This configuration prevents sliding from original position (because position which corresponds most to the original pattern is returning by tracker), but it can lead to small jumps and can lead to failures when feature point is getting deformed due to camera motion (perspective transformation, for example). Such configuration is used if Adjust Frames is set to 0.
  • Image created from current frame is sending as first image to the tracker. In this configuration pattern is tracking between two neighbor frames and it allows to deal with cases of high transformations of feature point but can lead to sliding from original position, so it should be controlled. Such configuration is used if Adjust Frames is set to 1.

If Adjust Frames is greater than 1, behavior of tracker would be like keyframes for tracks are creating every Adjust Frames frames and tracking between keyframed image and next image is used.

Speed can be used to control speed of sequence tracking. This option makes nothing with quality of tracking it's just helps to control if tracking happens accurate. In most of cases tracking is happens much faster than realtime playing and it's difficult to notice when track began to slide off from position. In such cases Speed can be set to Double or Half to add some delay between tracking two frames, so slide off would be noticed earlier and tracking process can be cancelled to adjust positions of tracks.

Frames Limit controls how many frames can be tracked when Track Sequence operator is called. So, each Track Sequence operation would track maximum Frames Limit frames. This also helps to notice slide off of tracks and correct them.

Margin can be used to make tracks disabled when they are becoming too close to image boundary. This slider controls "too close" in pixels.

KLT tracker options

KLT tracker is an algorithm used by default. It allows to track most kinds of feature points and it's motion. It's using pyramid tracking which works in the following way. Algorithm tracks larger image than pattern first to find general direction of motion. Then it tracks a bit smaller image to refine position from first step and make final position more accurate. This continues several times. Number of steps of such tracking is equal to Pyramid Level option and we tell that on first step tracking happens for highest pyramid level. So Pyramid Level=1 is equal to pattern itself, and each next level doubles tracking image by 2.

Search area should be larger than highest pyramid level and "free space" between search area and highest pyramid level defines how much feature can be moved from one frame to another.

Default settings should work in most of general cases, but sometimes pyramid level should be changes. For example, when footage is blurry, adding extra pyramid levels helps to track them.

This algorithm can fail in situations when feature point is moving in one direction and texture around feature point is moving in another direction.

SAD tracker options

On each step SAD tracker reviews the whole search area and finds pattern on second image which is mostly like pattern which is getting tracking. This works pretty fast, but can fail in several cases. For example, when there're another feature point which looks like tracking feature point appears in search area. In this case SAD will lead to jump of track from one feature to another.

Correlation defines threshold value for correlation between two patters which is still considering as successful tracking. 0 means there's no correlation at all, 1 means correlation is full.

There's one limitation: currently, it works for features 16x16 only.

Marker Panel

This panel contains numerical settings for marker position, pattern and search areas dimensions, offset fo anchor point from pattern center. All sliders are self-explainable.

Proxy / Timecode Panel
Proxy / Timecode Panel in clip editor

This panel contains options used for image proxies and timacodes for movies.

Proxy allows to display images with lower resolution in clip editor. This can be helpful in cases when tracking of 4K footage is happening on machine with small amount of RAM.

First four options are used to define which resolutions of proxy images should be build. Currently it's possible to build images 25%, 50%, 75% and 100% from original image size. Proxy size of 100% can be used for movies which contains broken frames which can't be decoded.

Build Undistorted means that proxy builder would also create images from undistorted original images for sizes set above. This helps to have faster playback of undistorted footage.

Generated proxy images are encoding using JPEG and quality of JPEG codec is controlled by Quality slider.

By default, all generated proxy images are storing to <path of original footage>/BL_proxy/<clip name> folder, but this loction can be set by hand using Proxy Custom Directory option.

Rebuild Proxy will regenerate proxy images for all sizes set above and regenerate all timecodes which can be used later.

Use Timecode Index can (and better be used) for movie files. Basically, timecode makes frame search faster and more accurate. Depending on your camera and codec different timecodes can make better result.

Proxy Render Size defines which proxy image resolution is used for display. If Render Undistorted is set, then images created from undistorted frame would be used. If there's no generated proxies, render size is set to "No proxy, full render" and render undistorted is enabled, undistortion will happen automatically on frame draw.

Tools available in reconstruction mode

Proxy / 2D Stabilization Panel in clip editor

There's one extra panel which is available in reconstruction mode - 2D stabilization panel.

This panel is used to define data used for 2D stabilization of shot. Several options are available in this panel.

First of all, it's list of tracks used to compensate camera jumps, or locaiton. It works in the following way: it gets tracks from list of tracks used for location stabilization and finds median point of all this tracks on first frame. On each other frame algorithm makes this point have the same position in screen coordinates by moving the whole frame. In some cases it's not needed to fully compensate camera jumps and Location Influence can be used in such cases.

Camera can also rotate a bit adding some tilt to the footage. There's <string>Stabilize Rotation</strong> option to compensate this tilt. Extra single track is needed to set for this and it works in the following way. On first frame of movie, this track is getting connected with median point of tracks from list above and angle between horizon and this segment is keeping constant along the whole footage. Amount of rotation applying on the footage can be controlled by Rotation Influence.

If camera jumps a lot, it'll be noticeable black areas near image boundaries. To get rid of this black holes, there's Autoscale option which finds smallest scale, applying which on the footage all black holes near image boundaries would eliminate. There's option to control maximal scale factor (Maximal Scale) and amount of scale applying on the footage (Scale Influence)