From BlenderWiki

[edit] Introduction to Advanced Concepts

Now that you've seen the basics for creating masks and keying shots, there are some additional concepts that you should know about. These include pre-processing your raw video footage, feathering the edges of your mask to make the images blend together better and correcting for color spill.

[edit] Video Pre-Processing

Sometimes the video footage that is used needs to be pre-processed to remove digital artifacts. Web-cams and DV video cameras record in YCbCr format, but compress the Cb and Cr channels more than the Y channel. This is because the eye is more sensitive to luminance differences than chroma differences and we don't notice the compression artifacts as much. If we did not correct for this, then when we pull a key (particularly with the chroma key node) you can get a blocky mask. There is a great explanation of what is happening at http://www.neopics.com/bluescreen/.

You will need to evaluate your source footage to know for sure. I use a group node that put together to split out the channels of the image (in different color spaces) and view each channel to see if I need to do any pre-processing.

[edit] Working with DV footage

In an effort to correct this condition when working with DV footage (thanks to whoever brought it to my attention in IRC) we filter the channels that need the additional information. How we filter it depends on the type of source material.

[edit] Solution for 4:1:1, horizontal blur

One solution is to separate the source into it's Y, Cb, and Cr channels then apply a horizontal 4 pixel blur on the Cb and Cr channels to smooth them out, artificially creating some of the missing information, then recombine them before trying pulling a green screen matte. This preprocessing step helps, but due to the nature of the missing information, you may need to adjust your final matte with the erode/dilate node and a blur to make it look right.

[edit] Solution for 4:2:0, horizontal and vertical blur

Based in the concepts of previous solution, we can see that for 4:2:0 the required blur is 2 pixels, both vertical and horizontally. This way the 2*2 blocks are smoothed and the effect is less visible. Also, some devices record in 4:1:0 which requires a horizontal and vertical blur of 4 pixels.

[edit] Solution for 4:2:2, horizontal blur

In you have a high end camera that uses this subsampling, following what we had already said and checking the proposed patterns linked at start of this section, the blur needed is only horizontal and with a size of 2 pixels.

[edit] Mask Adjustments

[edit] Erode/Dilate Node

The erode/dilate node works perfectly for growing or shrinking our mask. We want to shrink it down a little bit (about 1 pixel) and then run that through a blur node. The we shrink it down a little so that after the blur it is about the same size but with softer edges. I think this gives a much nicer blend of the foreground objects on the background since now the edge pixes of the foreground become slightly transparent and blend in with the background better.

[edit] Correcting for Color Spill

[edit] Color spill Node

When you're almost done with your composite, there is one last step you should consider. If there is a very scooby-dooish green glow coming from your foreground objects then you may need to correct for color spill. It happens when the light from green screen (in the original footage) reflects on to the foreground object. Good effects shots should be lit so that they don't get this reflection, but then if they were perfect, we wouldn't need this tutorial. The color spill node is very simple. Add it right before your processed shot combines with the background. Select the G button since it is the green color that we wish to reduce. You can use the B or R buttons (for blue and red respectively) if that is what your background color is. What this node does is check each pixel and if the green component is larger than the other components, it brings it down the average of the other two. The final effect is that the image no longer looks like it has a green tint.