From BlenderWiki

Correct Depth of Field (DoF) Blur for Scenes with Transparent Faces

Abstract

This article describes how to use the "Defocus" node in order to get a Depth of Field effect in a scene with partly transparent faces (like the branches in the images below) without suffering from the problem which can be seen in the images.

Blur with artifacts from zbuffer calculation+alpha transparent textures

Correct blur.

Although this tutorial might look pretty complex, it isn't. It's just quite detailed. However, I'd recommend this for experienced users only (if you're really new, it's probably not worth the effort).

Problem

The problem is simple: The Defocus node blurs the image accoring to additional information called Z-Buffer. This buffer is basically an extra image containing the distance from the camera to any pixel in the rending. Blender creates this buffer without parsing textures. It just uses the scene's geometry for that. Thus, if there's a face with a texture with transparency mapped to it (like the branches of the trees in my image), all of the face will have the depth the face has -- that's wrong. The areas of the face which are transparent should have the depth of the objects behind it.

Solution idea

This was not my idea, it comes from blenderartists.org forums, user »endi« -- thanks! The idea is, however, to use blender's "Mist" function to get a zbuffer with alpha support. Blender provides a "Mist" output for nodes containing mist depth information. How you can use this to make a correct blur will be described below.

How to use the Mist buffer for DoF blur I: Setting up the buffer

Setting up Mist

First, we need to set up mist to be used as a fake zbuffer. I don't know what settings are best, but you can see mine in the image:

• Use "Lin". It'll make mist progression linear, which seems to be most suitable as fake zbuffer for me.
• Set "Start" to something low.
• Set "Dist" to something which is below your scene, like, 150. If you set it too low, objects behind this limit will be completely blurred. You can enable "Mist" at your camera to check that limit.
• Set "Misi" to something aroung 0.2.

Note Mist doesn't need to be enabled for your scene. You'll just need the settings (don't click the Mist button).

Setting up the camera

Right-click your camera, and go to Editing (F9). Set Dof Dist to something like, say, 20. I don't think it does matter, because we will reconfigure this later anyway, but if you have entirely wrong values on several places, you know, it makes adjustments difficult.

Setting up the render layer

This is easily done. Just go to the render buttons, click the Render Layers tab and enable the Mist button located at the right.

How to use the Mist buffer for DoF blur II: Setting up the nodes

As the final step, we'll setup the nodes now. First, make sure you re-render your image; then make a setup like the one shown in the image at the right.

This setup is quite simple: Take the Mist buffer from the Render Layer, and connect it to a Brightness/Contrast node (from Color) menu. This will be used to adapt the values of the buffer to the ones of a "real" zbuffer. The output of this node you then connect to your Defocus node, together with the original image from the render layer. The output of the defocus node, of course, you connect to Composite.

Adapting the values of the fake zbuffer using the Brightness/Contrast node

We'll now use the Brightness/Contrast node to make the fake zbuffer look remotely like the original one. Otherwise, the results will be garbage.

• Step 1: Connect the Render layer's "Z" to the composite node's "Image" dot. The image you see in your render window will now appear completely white.
• Step 2: Click on the image with Left mouse button, and drag around a bit. A bar appears at the bottom; it reads something like

X: 123 Y: 123 | R: 255 G: 255 B: 255 A:255 | R: 13.543 G: 13.543 B: 13.543 A: 1.000 | Z: 13.543

• The only thing which is interesting for us is the numbers written in bold letters here. Just remember where they are roughly (usually about 15).
• Step 3: Now, connect the Brightness/Contrast node's output to the Composite "Image" node. Watch the image editor again, click and drag. The values you see at the places described above will be somewhere about 0.7, usually. What you now got to do is tweak the Contrast: value until they've rasied to the values you remember from the original zbuffer (about 15). This will mean, set the contrast to something like 99.
• Step 4: That's it! Connect the Brightness/Contrast node to the Defocus node again, connect the defocus node to Composite.

Now, you can just configure the Defocus parameters as you're used to. Additionally, you have the Brightness/Contrast node's "Brightness" slider to control the sharp point without re-rendering the layer. Values from -10 to 10 seem to be usefull.

If you have any questions, you'll maybe find me in IRC (scummos).