From BlenderWiki
Tutorial on doing object cutouts without booleans
The visual effect of boolean, cutting a piece out from one object using another, is often required. Using the boolean modifier is however always not optimal. The topology created by it is not always good, containing lots of triangles that produce shading and normal problems. Some mesh data, such as UVs, might also be lost or not correctly handled by using a generate modifier. Therefore a way of duplicating the effect without actually altering the base meshes is useful.
In this short description such a way is described. A hole into the material of the object will be made using a mask made by using dynamic paint, and the inner part of the intersection is made from a part of the other object, likewise using dynamicpaint to make a mask.
Both objects need a UV map, any mapping with sufficient padding between UV islands should do.
Using dynamic paint to make the masks used to mask the hole is pretty straight forward. One object must be assigned to be the canvas, other a brush. The brush has to use volume for painting, and use white as the color. Canvas should be in image format and in paint surface mode. The frames settings should be set to the timeframe the objects are supposed to be intersecting. Output the frames into a new directory.
The process of painting the mask has to be repeated to get the mask for the other object by swithcing the roles of the objcets. A way to rebake the masks should be left, so making linked copies of the objects and setting the second pair of brush and canvas on these copies is a good idea. These copies have to be either moved in the 3d space or sent to a linked scene to not interact with the original pair of objects. Make sure to bake the mask from the second pair into a new folder of image files.
Using these masks is pretty easy.
In cycles you should add an image node and set the settings for the image sequence. The black and white image is plugged into the value socket of a mix shader. The mix should have the actual shader node three of the object plugged to one socket and a transparency shader to the other. The same kind of mix is setup on both of the objects. All the combinations possible with the boolean modifier are achievable by changing the order of the shaders plugged to the mix nodes on the objects.
With blender internal renderer, the node materials can be added in the same manner as with cycles.
notes: TODO: images, links to relevant sections of wiki, cleanup, style change (avoidance of you-passive)
feel free to add suggestions or comments to my user page
