From BlenderWiki

Motion Tracking

Introduzione

Motion tracking è una nuova tecnica disponibile in Blender. E' ancora in via di sviluppo, e attualmente supporta solo operazioni di base per il camera tracking, ma è già pronto per essere usato in produzione.

Per iniziare

Motion tracking è disponibile nel corrente Trunk SVN e potrebbe essere inclusa nella versione 2.61 di Blender. E'abilitata di default per tutte le piattaforme e può essere usata "out-of-box".

Qui c'è una breve descrizione degli strumenti di motion tracking correntemente disponibili in Blender

Tracking supervisionato 2D

Non c'è alcun algoritmo comune che può essere utilizzato per tutti i tipi di pellicole, dei punti caratteristici e dei loro movimenti. Tali algoritmi potrebbero essere realizzati, ma il farlo potrebbe essere molto lento e portare a dei fallimenti, quindi l'unico modo per effettuare il monitoraggio 2D è quello di scegliere manualmente l'algoritmo di tracking e le sue impostazioni. I settaggi correnti di default potrebbero funzionare bene per la pellicola generica che non è molto sfocata e dove i punti funzione non sono sempre altamente deformati dalla prospettiva.

Migliorare il monitoraggio 2D è già nella nostra lista TODO, ma non è prioritario in questo momento. Se non siete sicuri di algoritmi e impostazioni e non si vuole per leggere questo documento, potete solo giocare con le impostazioni e trovarne una che funziona per voi.

Calibrazione manuale delle lenti usando il grease pencil e/o la grid

Tutte le registrazioni delle telecamere hanno il video distorto, non ci si può fare niente proprio per come funzionano le lenti. Per un accurato movimento della camera bisogna ricavare l'esatto valore della lunghezza focale e la "forza" di distorsione.

Attualmente, la lunghezza focale può essere ottenuta solo dalle impostazioni della fotocamera o da EXIF - non ci sono strumenti all'interno di Blender per stimarla. Ma ci sono alcuni strumenti che possono aiutare a trovare valori approssimativi per compensare la distorsione. Ci sono anche strumenti completamente manuali dove è possibile utilizzare la griglia la quale è influenzata dal modello di distorsione e definisce le linee rette delle celle deformate nella pellicola. Potete inoltre usare anche il grease pencil per questo - solo disegnare la linea che dovrebbe essere dritta sulla pellicola utilizzando un pennello poly linea e regolare i valori di distorsione per creare linee corrispondenti del grease pencil sul filmato.

Per calibrare la tua camera in modo più accurato l'unico modo è quello di utilizzare lo strumento di calibrazione della griglia da OpenCV. OpenCV sta usando lo stesso modello di distorsione, per cui non dovrebbe essere un problema.

Risoluzione del movimento della camera

Nonostante il fatto che non ci sia alcuna differenza nel risolvere il movimento della fotocamera e il movimento dell'oggetto da un punto di vista matematico, è attualmente supportata solo la risoluzione del movimento della camera. Ed essa ha ancora alcune limitazioni, come la non supportata risoluzione dei movimenti del treppiede o i movimenti del piano dominante dove tutte caratteristiche tracciabili appartengono ad un piano). E' previsto di risolvere queste limitazioni in futuro.

Strumenti di base per orientamento e stabilizzazione della scena

Dopo aver risolto il movimento, è necessario orientare la scena reale nella scena 3D per la composizione più conveniente. Ci sono strumenti per definire piano, origine della scena e assi X / Y per eseguire l'orientamento della scena.

Inoltre a volte è necessario stabilizzare il video proveniente dalla camera per creare un risultato finale che risulti più bello a vedersi. La stabilizzazione 2D è disponibile per stabilizzare il video che proviene dalla camera, essa può compensare salti e inclinazione della camera.

Nodi di base per comporre la scena nella pellicola reale

Alcuni nodi sono stati aggiunti nuovi al compositor per comporre la scena in filmati in modo semplice. Così ci sono i nodi per la stabilizzazione 2D, la distorsione e la undistortion che sono facili da usare.

Strumenti non implementati

Alcuni strumenti non sono ancora disponibili in Blender, ma sono nella nostra lista TODO. Quindi non c'è attualmente alcun supporto per cose come il filtraggio rolling shutter, la risoluzione del movimento dell'oggetto, motion capturing. Ma si può provare a ricavare con trucchi questa roba utilizzando le cose attualmente implementate.

Manuale

Movie Clip Editor

Quasi tutti gli strumenti di tracciamento del movimento sono concentrati nell'editor chiamato Movie Clip Editor. Attualmente non ha alcun strumento che non sia relativo al motion tracking, ma in funzione di esso può essere ampliato per essere usato per mascherare e così, ecco perché si chiama in questo modo più astratto che esula dal motion tracking.

Questo editor può essere trovato nella lista dei vari tipi di editor.

Quando commuterete al Movie Clip Editor l'interfaccia potrebbe cambiare nel seguente modo.

La prossima cosa che è logico fare è aprire il nuovo videoclip per cominciare a lavorarci su. Ci sono vari modi per farlo:

• Usare il pulsante Open dall'intestazione del movie editor
• Usare la voce Clip » Open del menu
• Usare la shortcut AltO>

Nel clip editor possono essere usati sia files filmati che sequenze di immagini. Se state usando sequenze di immagini c'è una limitazione sulla denominazione dei files: l'ultimo gruppo di numeri dovrebbe essere incrementato continuamente.

Una volta che il movie clip è caricato nel clip editor, dei pannelli extra appariranno nell'interfaccia.

Ci sono un sacco di nuovi strumenti che appaiono sullo schermo ed ecco una breve descrizione di tutti.

Prima di tutto, va ricordato, che il camera solver si compone di tre fasi ben distinte:

• Il tracking 2D del filmato
• Gli intrinsechi (lunghezza focale, coefficiente di distorsione) specifica/stima/calibrazione della camera
• Risoluzione della camera, orientazione della scena, ricostruzione della scena

Gli strumenti nel clip editor sono divisi a seconda del momento in cui vengono usati, per cui l'interfaccia non è inconfusionata con strumenti di orientamento scena quando solo il tracking 2D può essere fatto. Attualmente la categoria di strumenti visualizzati può essere cambiata usando la voce Mode del menu che si trova nell'intestazione dell'editor.

Ma quasi tutti gli operatori possono essere chiamati dai menu, quindi non è necessario cambiare il modo ogni volta che si vuole richiamare lo strumento che viene visualizzato nei pannelli su una diversa modalità di editor.

In modalità tracking sono visualizzati solo strumenti relativi al tracking e al camera solving. Gli strumenti del camera solving sono esposti qui perché è dopo aver risolto che più probabilmente vorrete effettuare il re-tracking delle tracce esistenti o piazzare nuovi brani per rendere la risoluzione più accurata.

Tools available in tracking mode

Markers panel

• Add Marker and Move operator places new marker at the position of mouse (which is under the button in this case, not ideal but it's just how things work) and then it can be moved to needed location. When it's moved to needed position, LMB can be used to finish placing new marker. Also, ↵ Enter and Space can be used to finish placing marker.

But it's faster to use CtrlLMB to place markers directly on the footage. This shortcut will place marker on place you've clicked. One more feature here: until you've released mouse button you can adjust marker position moving mouse and using track preview widget to control how accurate marker is placed.

• Detect Features operator detects all possible to features on current frame and places markers at this features. This operator doesn't take in account other frames, so it can place markers on features which belongs to moving object and if camera is turning away from this shot, no markers would be paced on frames after camera moved away.

There are several properties for this operator:

Placement is used to control where to place marker. By default, they'll be added on the whole frame, but you can also outline some interesting for detecting areas with grease pencil and place markers only inside outlined area. That's how "Inside Grease Pencil" placement variant works. You can also outline absolutely not interesting areas (like trees, humans and so) and place markers outside of this areas. That's how "Outside Grease Pencil" placement variant works.

Margin controls distance from image boundary for created markers. If markers would be placed too close to image boundary, they'll fail to track really soon and they should be deleted manually. To reduce amount of manual cleaning up this parameter can be used.

Trackness limits minimal trackness for placing markers. This value came from feature detection algorithm and basicly it means: low values means most probably this feature would fail to track very soon, high value means it's not much such track. Amount of markers to be added can be controlled with this value.

Distance defines minimal distance between placed markers. it's needed to prevent markers placed too close to each other (such placement can confuse camera solver).

• Delete Track is quite self-explaining operator which deletes all selected tracks.

Track panel

• First row of buttons is used to perform tracking of selected tracks. Tracking can happen (in order of buttons):
  • Backward one frame
  • Backward along the sequence
  • Forward along the whole sequence
  • Forward one frame

This operators depends on settings from Tracking Settings panel which would be described later.

If during sequence tracking algorithm returned failure of tracking some markers, they'll be disabled and tracking will continue for rest of markers. If algorithm returns failure when tracking frame-by-frame, marker wouldn't be disabled and most possible position of feature on new frame would be sued.

• Clear After deletes all tracked and keyframed markers after current frame for all selected tracks.
• Clear Before deletes all tracked and keyframed markers before current frame for all selected tracks.
• Clear clears all markers except current form all selected tracks.
• Join operator joins all selected tracks into one. Selected tracks shouldn't have common tracked or keyframed markers at the same frame.

Solve panel

Camera Motion operator solves motion of camera using all tracks placed on the footage and two keyframes specified on this panel. There are some requirements:

• There should be at least 8 common tracks on the both of keyframes
• It should be well noticeably parallax effect between this two keyframes

If everything came smooth during solve, average reprojection error would be reported to the information space and to clip editor header. Reprojeciton error means average distance between reconstructed 3D position of tracks projected back to footage and original position of tracks. Basically, reprojection error below 0.3 means accurate reprojection, 0.3-3.0 means quite nice solving which still can be used. Values above 3 means some tracks should be tracked more accurate or that incorrect values for focal length or distortion coefficients were set incorrectly.

Refine option specifies which parameters should be refined during solve. Such kind of refining is useful when you aren't sure about some camera intrinsics and solver would try to find best parameter for that intrinsics. But you still have to know approximate initial values - it'll fail to find correct values if they were set completely incorrect initially.

Cleanup Panel

This panel contains single operator and it's settings. This operator cleans up bad tracks: tracks which aren't tracked long enough or which failed to reconstruct accurate. Threshold values can be specified from slides below the button. Also, several actions can be performed for bad tracks:

• They can be simply selected
• Bad segments of tracked sequence can be removed
• The whole tracks can be deleted

Clip Panel

This panel currently contains the single operator Set as background which sets currently editing clip as camera background for all visible 3D viewports. If there's no visible 3D viewports or clip editor is opened in full screen, nothing will happen.

Properties available in tracking mode

Grease Pencil Panel

It's a standard grease pencil panel where new grease pencil layers and frames can be controlled. There's one difference in behavior of grease pencil from other areas - when new layer is created "by-demand" (when making stroke without adding layer before this) default color for layer would be set to pink. It makes stroke easy to notice on all kinds of movies.

Track Panel

First of all, track name can be changed in this panel. Track names are used for linking tracking data to other areas like follow track constraint.

Next thing which can be controlled here is marker's enabled flag (using button with eye icon). If marker is disabled, it's position wouldn't be used neither by solver nor by constraints.

Button with lock at the right of button with eye means track is locked. Locked tracks means it can't be edited at all. This helps to prevent accidental changes to tracks which are "finished" (tracked accurate along the whole footage).

Next widget placed on this panel is called "Track Preview" and it displays content of pattern area. This helps to check how good tracking is happening (control there's no slides from original position) and also helps to move track back to needed position. Moving of track can happen directly from this widget by mouse slide.

If anchor is used (position on image which is tracking is differ from position which is used for parenting) preview widget will display area around anchor position. Such configuration helps for masking some things when there's no good feature at position where mask corner should be placed. Details of this technique would be written later.

There's small area below the preview widget which can be used to enlarge vertical size of preview widget (area is highlighted by two horizontal lines).

Next setting is channels control. Tracking happens in grayscale space, so it can be not enough contrast between feature and background to perform accurate tracking. In such cases disabling some color channels can help.

The last thing is custom color and preset for it. This settings overrides default marker color used in clip editor and 3D viewport and it helps to distinguish different type of features (for example features on far plane or near plane and so). Color is also can be used for "grouping" tracks so the whole group of tracks can be selected by color using Select Grouped operator.

Camera Data Panel

This panel contains all settings of camera used for screening movie which is currently editing in clip editor.

First of all, predefined settings can be used here. New presets can be added or unused presets can be deleted. But such settings as distortion coefficients and principal point aren't included into presets and should be filled in even if camera presets were used.

• Focal Length is already self-described, it's a focal length with which movie was shoot. It can be set in millimeters and pixels. In most cases focal length is given in millimeters, but sometimes (for example in some tutorials in the Internet) it's given in pixels. In such cases it's possible to set it directly in units it's known in.
• Sensor Width is a width of CCD sensor in camera. This value can be found in camera specifications.
• Pixel Aspect Ratio is a pixel aspect of CCD sensor. This value can also be known from camera specifications, but can also be guessed. For example, you know that footage should be 1920x1080 but images itself are 10280x1080. In this case pixel aspect is:

1920 / 1280 = 1.5

• Optical Center is an optical center of lens used in camera. In most cases it's equals to image center, but it can be differ in some special cases. Check camera/lens specifications in such cases. To set optical center to center of image, there's Center button below sliders.
• Undistortion K1, K2 and K3 is a coefficients used to compensate lens distortion happened on movie shooting. Currently this values can be tweaked by hand only (there's no calibration tools yet) using tools available in Distortion mode. Basically, just tweak K1 until solving would be most accurate for known focal length (but also take grid and grease pencil into account to prevent "impossible" distortion)

Display Panel

This panel contains all settings which controls things displayed in clip editor.

• Pattern can be used to disable displaying of rectangles which are correspond to pattern areas of tracks. In some cases it helps to make clip view more clear and check how good tracking is.
• Search can be used to disable displaying of rectangles which are correspond to search areas of tracks. In some cases it helps to make clip view more clear and check how good tracking is. Search areas for selected tracks only would be displayed.
• Pyramid makes the highest pyramid level be visible. What pyramid itself is would be described later in Tracking Settings panel section, but basically it helps to determine how much track is allowed to move from one frame to another.
• Track Path and Length controls displaying of path of tracks. So way in which track s are moving can be visible looking at the only one frame. It helps to determine if track jumps from it's position or not.
• Disabled Tracks makes possible to hide all tracks which are disabled on current frame. This helps to make view more clear to control if tracking happens accurate enough.
• Bundles makes sense after solving movie clip and it works in the following way: solved position of each track gets projected back to movie clip and displayed as small point. Color of point depends on distance of projected coordinate and original coordinate: if they are close enough point would be green, otherwise it'll be red. This helps to find tracks which weren't solved nicely and need to be tweaked.
• Track Names ans Status displays such information as track name and status of track (if it's keyframed, disabled, tracked or estimated). Names and status for selected tracks is displayed.
• Compact Markers. Way in which markers are displayed (black outline and yellow foreground color) makes tracks be visible on all kind of footages (both dark and light). But sometimes it can be annoying and this option will make displaying of marker in more compact manner - outline would be replaced by dashed black lines drawing on top of foreground, so markers areas would be 1px thick only.
• Grease pencil means if grease pencil strokes are allowed to be displayed and made.
• Mute changes displaying on movie frame itself with black square, It helps to find tracks which are tracked inaccurate or which weren't tracked at all.
• Grid (available in distortion mode only) displays grid which is originally orthographic but was affected by distortion model. This grid can be used for manual calibration - distorted lines of grids are equal to straight lines in the footage.
• Manual Calibration (available in distortion mode only) applies distortion model for grease pencil strokes. This option also helps to perform manual calibration. More detailed description of this process would be made later.
• Stable (available in reconstruction mode only). This options makes displaying frame be affected by 2D stabilization settings. It's only preview option which doesn't actually changes footage itself.
• Lock to Selection makes editor be displaying selected tracks on the same screen position along the whole footage during playback or tracking. This option helps to control tracking process and stop it when track is starting sliding off or when it jumped.
• Display Aspect Ratio changes aspect ratio for displaying only. It does not affect on tracking or solving process.

Tracking Settings Panel

Common options

This panel contains all settings for 2D tracking algorithms. Depending on which algorithm is used, different settings would be displayed, but there are also few common for all trackers settings:

Adjust Frames controls which patterns are getting tracking, if be more precious, pattern from which frame is getting tracked. Here's an example which should make things more clear.

Tracker algorithm receives two images inside search area and position of point to be tracked in first image. And tracker tries to find position of point from first image on second image.

Now, how tracking of sequence happens. Second image is always image created from frame at which position of marker isn't known (next tracking frame). But different first image can be send to tracker. Most commonly used combinations:

• Image created from frame on which track was keyframed. This configuration prevents sliding from original position (because position which corresponds most to the original pattern is returning by tracker), but it can lead to small jumps and can lead to failures when feature point is getting deformed due to camera motion (perspective transformation, for example). Such configuration is used if Adjust Frames is set to 0.

• Image created from current frame is sending as first image to the tracker. In this configuration pattern is tracking between two neighbor frames and it allows to deal with cases of high transformations of feature point but can lead to sliding from original position, so it should be controlled. Such configuration is used if Adjust Frames is set to 1.

If Adjust Frames is greater than 1, behavior of tracker would be like keyframes for tracks are creating every Adjust Frames frames and tracking between keyframed image and next image is used.

Speed can be used to control speed of sequence tracking. This option makes nothing with quality of tracking it's just helps to control if tracking happens accurate. In most of cases tracking is happens much faster than realtime playing and it's difficult to notice when track began to slide off from position. In such cases Speed can be set to Double or Half to add some delay between tracking two frames, so slide off would be noticed earlier and tracking process can be cancelled to adjust positions of tracks.

Frames Limit controls how many frames can be tracked when Track Sequence operator is called. So, each Track Sequence operation would track maximum Frames Limit frames. This also helps to notice slide off of tracks and correct them.

Margin can be used to make tracks disabled when they are becoming too close to image boundary. This slider controls "too close" in pixels.

KLT tracker options

KLT tracker is an algorithm used by default. It allows to track most kinds of feature points and it's motion. It's using pyramid tracking which works in the following way. Algorithm tracks larger image than pattern first to find general direction of motion. Then it tracks a bit smaller image to refine position from first step and make final position more accurate. This continues several times. Number of steps of such tracking is equal to Pyramid Level option and we tell that on first step tracking happens for highest pyramid level. So Pyramid Level=1 is equal to pattern itself, and each next level doubles tracking image by 2.

Search area should be larger than highest pyramid level and "free space" between search area and highest pyramid level defines how much feature can be moved from one frame to another.

Default settings should work in most of general cases, but sometimes pyramid level should be changes. For example, when footage is blurry, adding extra pyramid levels helps to track them.

This algorithm can fail in situations when feature point is moving in one direction and texture around feature point is moving in another direction.

SAD tracker options

On each step SAD tracker reviews the whole search area and finds pattern on second image which is mostly like pattern which is getting tracking. This works pretty fast, but can fail in several cases. For example, when there're another feature point which looks like tracking feature point appears in search area. In this case SAD will lead to jump of track from one feature to another.

Correlation defines threshold value for correlation between two patters which is still considering as successful tracking. 0 means there's no correlation at all, 1 means correlation is full.

There's one limitation: currently, it works for features 16x16 only.

Marker Panel

This panel contains numerical settings for marker position, pattern and search areas dimensions, offset fo anchor point from pattern center. All sliders are self-explainable.

Proxy / Timecode Panel

This panel contains options used for image proxies and timacodes for movies.

Proxy allows to display images with lower resolution in clip editor. This can be helpful in cases when tracking of 4K footage is happening on machine with small amount of RAM.

First four options are used to define which resolutions of proxy images should be build. Currently it's possible to build images 25%, 50%, 75% and 100% from original image size. Proxy size of 100% can be used for movies which contains broken frames which can't be decoded.

Build Undistorted means that proxy builder would also create images from undistorted original images for sizes set above. This helps to have faster playback of undistorted footage.

Generated proxy images are encoding using JPEG and quality of JPEG codec is controlled by Quality slider.

By default, all generated proxy images are storing to <path of original footage>/BL_proxy/<clip name> folder, but this loction can be set by hand using Proxy Custom Directory option.

Rebuild Proxy will regenerate proxy images for all sizes set above and regenerate all timecodes which can be used later.

Use Timecode Index can (and better be used) for movie files. Basically, timecode makes frame search faster and more accurate. Depending on your camera and codec different timecodes can make better result.

Proxy Render Size defines which proxy image resolution is used for display. If Render Undistorted is set, then images created from undistorted frame would be used. If there's no generated proxies, render size is set to "No proxy, full render" and render undistorted is enabled, undistortion will happen automatically on frame draw.

Tools available in reconstruction mode

There's one extra panel which is available in reconstruction mode - 2D stabilization panel.

This panel is used to define data used for 2D stabilization of shot. Several options are available in this panel.

First of all, it's list of tracks used to compensate camera jumps, or locaiton. It works in the following way: it gets tracks from list of tracks used for location stabilization and finds median point of all this tracks on first frame. On each other frame algorithm makes this point have the same position in screen coordinates by moving the whole frame. In some cases it's not needed to fully compensate camera jumps and Location Influence can be used in such cases.

Camera can also rotate a bit adding some tilt to the footage. There's <string>Stabilize Rotation</strong> option to compensate this tilt. Extra single track is needed to set for this and it works in the following way. On first frame of movie, this track is getting connected with median point of tracks from list above and angle between horizon and this segment is keeping constant along the whole footage. Amount of rotation applying on the footage can be controlled by Rotation Influence.

If camera jumps a lot, it'll be noticeable black areas near image boundaries. To get rid of this black holes, there's Autoscale option which finds smallest scale, applying which on the footage all black holes near image boundaries would eliminate. There's option to control maximal scale factor (Maximal Scale) and amount of scale applying on the footage (Scale Influence)