From BlenderWiki

Mask Transparency

Mask Transparency is a special kind of transparency (to see the other two modes, visit: Raytraced Transparency). Instead of making objects transparent/translucent, it hides the geometry behind it.

Show Background

Our first example is Manual-Materials-Properties-Mask_Transparency-00.blend‎. The "Blender" text has its Transparency set to Mask with an Alpha of zero, and we set up a fancy procedural background in the World settings. When we render, the text masks off all objects going all the way to the background:

Notice the background peeking out to the right, by the backdrop---but also notice that it's piercing through everything where our "Blender" text is!

Render Layers

Here is our next example, which uses the Compositor to selectively overlay Blender-rendered images over a photo. The key elements are:

1. The hand was modeled, and set to a Transparency type Mask with Alpha 0.
2. The Compositor uses an AlphaOver to apply the render over the photo.
3. There is a fake "shadow-catcher" geometry, which has had these settings:
   • Shadow > Shadows Only was checked, plus "Shadows and Distance" was changed to "Shadows Only"
   • Traceable turned off (so it doesn't cast shadows)

Here is the .blend: Manual-Materials-Properties-MaskTransparency.blend

Before:

After

Animation

These techniques do not only have to apply to static images. You can get some amazing video effects.

Here is such a video: Sky Columns of Mystery.

And here is the .blend for that: Manual-Material-Properties-MaskTransparency-SkyColumns.blend An additional interesting aspect of that .blend is that it uses the VSE to composite Scenes, not Images. There is no intermediate render step, and Masking works to pierce through one scene to show another without a problem.