Reference/Release Notes/2.64/Sculpting/Implementation Notes
Sculpt masking implementation notes
Paint masks are backed by two new CustomData layers, CD_PAINT_MASK and CD_GRID_PAINT_MASK. These are analogous to Mvert and MDisp layers, respectively. In both layer types, each mask element is stored as a float, clamped always to between zero and one.
It is now assumed that the mesh always has a mask layer if it's in sculpt mode. This makes sense from a UI perspective; the user shouldn't have to take any action to start using masks.
Multiresolution masks are backed by a CD_GRID_PAINT_MASK. Unlike MDisps, these values are absolute (rather than relative displacements from the subdivided values.) In order for masks to subdivide and update smoothly, CCGSubSurf was modified to support subdivision of a variable number of floating-point layers. For subsurf, the number of layers stays at three (for coordinates), but for multires, the number of layers is increased to four so that mask values can be interpolated.
PBVH drawing in the GPU_buffers code assumes a mask layer is present. Because mask values are scalar, and OpenGL does not support single-element colors in the fixed-function pipeline, the multires VBO code now uses the same VertexBufferFormat structure as mesh VBO (the structure now has space for color, represented as three chars.)