From BlenderWiki

Mask Tools

Abstract

Currently there's no way to work with masks in Blender. It makes compositing of rendered scene into real footage difficult and requires tangled setups of render layers and hooks modifiers.

The very high goal of this project is to introduce more clear and comfortable way of interacting with masks in Blender in a way like:

• Create mask in editor space using different tools.
• Animation system, drivers, tracking data can be used to drive masks over the time.
• Mask can be reused all over.
• Mask is flexible to be used for FullHD final render and proxied footage for the preview.

General Info

Mask is a grayscale raster image which is created from vector image. Artists interacts with vector image to create mask with needed shape and then it automatically gets rasterized when using in compositor or wherever else.

User interacts with masks in the following editors:

• Movie Clip Editor
• Image Editor
• Compositor
• Sequencer

Clip Editor and Image editor are used to edit masks, Compositor and Sequencer are just using already created mask.

Masks has got own ID block and can be accessed form all over.

Editing Masks

Masks can be edited in either Image Editor or Movie Clip Editor. Movie Clip Editor allows to link mask to motion tracking data to drive the whole mask or it's points using data came from tracker.

Editing of masks happens in a way similar to editing bezier curves or (pathes) in GIMP: control points are adding to define general shape of mask, handles of different types are used to smooth bends. This allows to define only few points of mask and make it following needed object on the footage smoothly. Internally control points and handles coordinates are stored in normalized 0..1 space which allows masks to be reused for any kind of resolution.

To prevent tools mess, special mode called Masking is added to Clip and Image editors. When Maskign mode is enabled, almost all Clip/Image specific tools and properties are hidden and only tools needed for interactig with masks are displayed.

It's possible to control feather of mask, including a way to define non-linear feather. Linear feather is controlled by a slider, non-linear feather is controlled in the same curve-based way to define feather falloff.

Using Masks

When mask is used in compositor or sequencer, it's getting automatically converted to raster and further they're used as regular grayscale image.

In compositor there's special input node needed mask can be picked. Output socket of this contains rasterized image created from mask. This rasterized image can be used as input for other nodes such as Invert, Multiply, Mix nodes and so on.

By default, render resulution is used to rasterize mask for Mask input node. If RGBA buffer is connected to this node, it's resolution would be used for rasterization.

In the sequencer it can be special type of strip or effect strip.

Accessing masks

Using masks in motion-tracking workflow isn't the only possible usage of masks: they can also be used for things like rotoscoping. So, mask isn't necessarily comes together with MovieClip datablock -- it can be used against rendered scene to achieve some result. Hence, mask as independent ID block allows to access them in the same way from every area of Blender.

When Movie Clip Editor is switched to Masking mode, it's possible to choose mask to edit and choose clip to be used as a backdrop and as "source" of motion tracking information for pinning mask curves' points to markers.

Define Mask Shape

Ways to define mask shape:

• LMB is used to place new control point and define handles orientation (click to place control point, click followed with slide to place new control point and set smoothness for it).
• Existing control points can be dragged.
• Mask can be created from grease pencil strokes.
• Ctrl + LMB is used to define feathering outline curve.

Mask can be created from non-closed curve (which is useful to mask out linear objects like wires, hairs and so).

Drive Masks

Masks can be driven over the time so it'll be following some object from the footage (e.g. running actor), This can be done in several ways:

• Keyframed animation of control points using standard animation system (F-Curves). This can be useful when there's no enough quality, or no good feature points in the footage.
• Control points can be driven by regular drivers system.
• Control point can be pinned to some motion track (track created by motion-tracking tools). This way is the main way to interact with masks in motion-tracking workflow.
• The whole mask can be driven instead of individual control points.
• Masks can be parented to each other, so driving one mask would imply driving parented masks.

Some other ideas

There are plenty of ways to make masks totally rock, but some of them requires redesign of other areas of Blender. Some of such ways are:

• Create masks from node editor.
• Mask size. With current compositor it's not so obvious to guess which resolution should be used to rasterize mask. Canvas-based compositor should make it possible to automatically find out needed resolution to rasterize mask.
• Motion blur (see discussion below).

Code: Data Structures

Data Structures Diagram

Here's brief diagram of data structures needed to support all features of masks listed above. Also main relations between data structures is displayed.

Data structures and relation for masks

Data Structures

Main data structure which defines mask:

typedef struct Mask {
	ID id;
	struct AnimData *adt;
	ListBase splines;	/* splines which defines this mask */
	/* ...  */
} Mask;

Data structure which hols all data needed for parenting:

typedef struct MaskParent {
	int parent_type;      /* type of parenting */
	ID *parent;           /* ID block of entity to which mask/spline is parented to
	                       * in case of parenting to movie tracking data set to MovieClip datablock */
	char sub_parent[64];  /* sub-entity of parent to which parenting happened
	                       * in case of parenting to movie tracking data contains name of track */
	float offset[2];      /* offset from parent position, so object/control point can be parented to a
	                       * motion track and also be animated (see ZanQdo's request below)  */
} MaskParent;

Each mask is a set of splines which are getting rasterized using some algorithm. Here's data structure which defines spline itself:

typedef struct MaskSpline {
	int flag;
	float radius;
	MaskSplinePoint *points;          /* points which defines spline itself */
	MaskSplinePoint *feather_points;  /* points which defines feather of spline */
	MaskParent parent;
} MaskSpline;

Flag in MaskSpline contains such bits as if spline is closed or not (maybe something else in future).

Points used for spline itself and it's feather are the same data type:

typedef struct MaskSplinePoint {
	BeztTriple bezt;    /* actual point position, handles, etc.. */
	maskParent parent;  /* parent of point */
	int origindex;
} MaskSplinePoint;

Origindex is used to "link" points of feather to points of spline. It's needed because feather has got the same points as spline itself and until point is getting disconnected from it, such tools as transform should act on both of spline and it's feather.

Discussion

Sobotka 09:43, 11 January 2012 (CET)

Two points:

• Feathering. Should allow for a value that can be adjusted for feathering both negative (outline of spline inwards) to positive (outline of spline outwards.)
• Motion blur. A toggle that uses the currently set shutter speed (Shouldn't this be a global value anyways?) and calculate the tweening distance according to shutter speed.

xglasyliax 16:58, 11 January 2012 (CST)

Regarding point #1 above. The way I envision handling the double edge part is to pull out a 2nd curve from the first. Perhaps holding a SHFT or CTRL key while moving a vert would pull out a secondary vert, theoretically "connected" to the original. If you pull out a secondary set of points, you can define an outer edge curve very easily, and it's easy to both see and animate, just like a regular curve. The secondary curve would then connect back to the primary curve at any vert which has no outer edge vert of it's own.

ZanQdo 16:58, 17 January 2012 (CST)

About driven/tracked points: Manual animation should be possible on top of them, as an animation layer or parent behavior in order to fine tune the results

About keyframing: I think we need to decide the method of keying. Full blocked seems to be usually the best method, at least per bezier, meaning keyframes are inserted in *all* CVs of a bezier

About layering: Would like to see some talk about how to handle multiple layers of masks, their opacities, their order and their subtractive vs additive capabilities