11
The Matte Effects
The primary focus of the Matte effects is transparency; or more specifically, the alpha channel of a layer. They are usually used in conjunction with keying or rotoscoping. The Matte effects restore holes in a layer (typically caused by keying), and can also eat away at the edges of a layer.
In the first edition of this book, we looked at the two original effects in this category: Matte Choker and Simple Choker. Since After Effects CS5, we have two new additions—Mocha Shape and Refine Edge. Mocha Shape isn’t really a standalone effect—it’s designed to be used with Mocha, which is a separate app included with After Effects CS6. So we won’t be looking at that here.
But the new gem of this category is the powerhouse Refine Matte effect. This effect contains options that are part of the Rotobrush feature. But these options were so amazing, they were made a separate effect (Refine Matte). And although Matte Choker and Simple Choker still work great, I always go first to Refine Matte. It is almost frighteningly intelligent, and has fixed many problems that I thought impossible to fix.
3rd Party Matte Assistance
Since writing the first edition of this book, I’ve since discovered (and subsequently fallen in love with) the Key Correct tools from Red Giant Software (redgiantsoftware.com). Key Correct is a suite of 15 tools designed purely to help with refining mattes, pulling better keys, and getting a better overall composite. It seems like I use at least one Key Correct effect every single time I do compositing work, and I can’t imagine life without them. If you do a lot of this kind of work, I highly suggest taking a look at what Key Correct can offer you.
Here’s a simple breakdown of these effects compared. Simple Choker is the fastest to render and easiest to understand. Matte Choker is the hardest to understand but it gives more control than Simple Choker and renders very quickly. Refine Matte gives the best results in most cases and certainly does all kinds of great things the others can’t do, but it is by far the slowest to render. And while it’s more intuitive than Matte Choker, it’s much more annoying to experiment with because everything takes so much longer. Still, Refine Matte is my go-to native effect when working with mattes.
For Matte Choker and Simple Choker, there are some common terms that we should look at before proceeding. The first concept is that of spreading. Spreading is when you increase the size of the alpha channel (matte), usually by filling in holes inside the layer. The other concept we’ll use is that of choking. Choking refers to shrinking, or constricting the matte, usually to get rid of a halo of pixels around a subject.
The Matte Choker Effect
The Matte Choker effect is used to both spread and choke a matte. If you open the Matte Choker.aep project found in the Chapter 11 folder of the exercise files, you’ll see a simple project. Here, I’ve taken a photo of me, your humble author, in front of a green screen and keyed out the background using the Color Key effect. But, as is the case with keying on occasion, we’ve actually created some holes in our subject.
Figure 11.1
A close up of the subject as seen in the Matte Choker.aep project. The bright blue areas are holes in our layer.
We’re going to use the Matte Choker effect to restore these areas back to full opacity, and to fine tune the edge of the matte as well. I’ve created a bright blue solid and placed it below the subject in the layer stack. This will help us identify and fix problems in our key.
The first thing we need to realize about the Matte Choker effect, is that it is made up of two main sections, each with the same three controls. The three controls are Geometric Softness, Choke, and Gray Level Softness. The first set of these controls will be used to spread (restore) the matte. The second set will be used to shrink (constrict) the matte.
Figure 11.2
The parameters of the Matte Choker effect.
One of the beefs I have with this effect is that the default settings aren’t the best defaults, as I see it. No matter what your settings are when using this effect, it’s almost guaranteed that they will be very different from the default values.
Checking Your Matte
Often, checking mattes against the default black background is a bad idea. Mattes can be tricky little devils. Many times, you won’t be able to see problems with a matte until you either view the alpha channel (using the Show Channels and Color Management Settings button at the bottom of the Composition, Footage, and Layer panels), checking the matte against the transparency grid (also a button at the bottom of the Monitor panel), or by previewing your matte against different color backgrounds. Sometimes, it takes a combination of all of these before I catch a problem.
Let’s look first at Geometric Softness 1. Geometric Softness controls how smooth the edges of the matte are, and it also shrinks the matte a little. Take this value to about 3. A higher Geometric Softness might look better here, but taking this value too high will limit our ability to properly choke back the matte later.
Next is the Choke 1 value. A positive Choke value chokes the matte, while a negative Choke value spreads the matte. Since we’re spreading the matte with the first group of controls, take the Choke value to –125. This allows the holes in our subject
Figure 11.3
After spreading the matte.
to become opaque again. Finally we come to the Gray Level Softness 1 value. This controls the softness of the edge of the matte. Take this value to about 7%.
Gray Level Softness 2 Default
The default value for the Gray Level Softness 2 property is set at an almost ridiculous 100%. In many cases, this will prevent you from seeing any changes you make to the second set of properties. If you’re just fiddling with the second group of properties, be sure to first reduce the Gray Level Softness parameter so you can see what you’re doing.
Our results already look much better. And, although I still look cheesy, I don’t look swiss cheesy—the holes on my layer have been filled. But in order to make things even better, we need to use the second set of these controls in the Matte Choker effect to choke back the matte.
Essentially, what I did in this case, is to duplicate the values of the first set again, except that the Choke 2 value is set to a positive 125 to choke the matte. This means that Geometric Softness 2 is set to 3, and Gray Level Softness 2 is set to 7%. The iterations value can be increased also, which is like adding another round of spreading and choking.
Figure 11.4
The final result after spreading and choking the matte.
The Refine Matte Effect
The Refine Matte effect was added with the Rotobrush tool back in After Effects CS5. It should be noted that the same tools exist in both. The Refine Matte effect is simply the best matte refinement tool that exists in After Effects. It’s almost magical what it can do to a key, including removing a background from a moving subject with motion blur. I don’t know of another tool that can do that.
We’re going to look at a simple example here. Open the Refine Matte.aep project from the Chapter 11 folder of the exercise files. Go to the Refine Matte START comp, where I have a shot from my short film GODLIZZA. I keyed this footage with Keylight after applying color correction and all that stuff. But it’s still just rough. I really don’t want to contract the matte anymore because it would eat away too much of the image before it would smooth it out. Most of the problems are around the hair of both kids. I could (and should probably) make a duplicate layer with a separate mask and rotoscope that. But it’s just that I’m so lazy.
Figure 11.5
The original material after being color corrected and keyed.
So let’s apply the Refine Matte effect to this. Wow. Even the default settings are an improvement over our original. I should point out that a lot of these settings are things that we’ve seen before—Smooth, Feather, Choke, etc. But the magic here is that they’re not sensitive at all, as opposed to these controls in other effects.
I’m going to increase Smooth to 2.5 (Smooth is actually the most sensitive parameter here, and you have to be a bit careful). Let’s bump up Feather to 50%, and Choke up to 85% (yep, 85%). You would think that settings that high would destroy our footage. But such is not the case. Refine Matte was made for fine tuning adjustments, just like this.
In our case, we don’t need it look for motion blur, so we can just deselect that option. We do have a wee bit of green spill left over. Making sure that Decontaminate Edge Colors is selected, take the Increase Decontamination Radius to 3.0. This increases the area that is decontaminated. Like Smooth, this is another control where a little goes a long way, so be aware of that.
Also with this effect, be aware that as great and as helpful as it usually is, the Use Motion Blur and Decontaminate Edge Colors options can occasionally ruin this effect. If you’re noticing weird artifacts and visual glitches, try turning off these options and see if your problems aren’t fixed. But remember that they can be life-savers in some cases also.
The Simple Choker Effect
Although the Simple Choker does, in fact, live up to its name and is quite simple, it can be useful in a variety of circumstances, as we’ll see. There is just one parameter here—Choke Matte. Negative Choke Matte values spread the matte, while positive values choke the matte. Obviously there’s less control here than with the Matte Choker or Refine Matte effects, but there’s also a lot less fuss. Let’s open up the Simple Choker.aep project from the Chapter 11 folder.
Here we see the example we looked at previously. What I did here is try to simulate all the work we did with the Matte Choker effect. The Simple Choker appears to round and soften the matte edge slightly already. So, I applied the Simple Choker effect, and took the value to negative 3.6, because negative values spread the matte. This filled in all the holes. Then I selected the effect in the Effect Controls panel, and pressed Ctrl+D(Win)/Cmd+D(Mac) on my keyboard to duplicate the effect. On the duplicate, I changed the value to a positive 2. This replicates what we did with the Matte Choker effect because it spreads, then chokes.
Next, go over to the 3D edge cleanup composition. This is a file that we looked at in the beginning of the book in our coverage of the 3D effects. You might have noticed then that these 3D files have terrible aliasing on the edges. That’s just the nature of 3D files when imported into After Effects this way. The Simple Choker effect can actually help us smooth out those ugly rough edges.
Figure 11.6 shows the result of our 3D file with the ID Matte effect applied, and the car isolated. Even though I’ve already tried here to smooth the edges of the matte with ID Matte’s Feather property, they still look awful.
So what we’re going to do is apply the Simple Choker effect. Remember that the Simple Choker effect also smoothes the matte in addition to spreading or choking it. In many cases, that can be a detriment as corners get rounded. But in this instance, a little extra smoothness is just what the doctor ordered. Apply the Simple Choker effect and take the Choke Matte value to a positive 1.5. Even zoomed in at 200%, the results look pretty good.
What else can the Simple Choker effect be used for? I find it handy for those times when I’ve imported a layered Photoshop file that contains objects I cut out of a background. Sometimes, if I’m working in a hurry in Photoshop, my selections might not be
Figure 11.8
The blocky edges on the isolated car are seemingly impossible to fix. Feathering the edges just makes a bigger mess.
Figure 11.9
The results after applying the Simple Choker effect.
the best. That means that there will probably be some extra pixels on the edges of my imported Photoshop layers. A quick application of the Simple Choker effect can often fix the problem.
I’ve heard it suggested that the Simple Choker effect can also compensate for an incorrect alpha interpretation. For example, if an object you import has a premultipled alpha channel, and you import it as straight, it will have a halo around it. The suggestion was to remove the halo with the Simple Choker effect. However, I don’t recommend this course of action. You will almost always get significantly better results from interpreting the alpha channel correctly in the Interpret Footage dialog box (which now has a nifty shortcut button in the bottom left hand corner of the Project panel in After Effects CS4 and later), rather than by using effects to “interpret” the alpha.
With a lot of these simple tasks, I might still prefer to use the Simple Choker effect, rather than calling on the slow rendering, big guns of Refine Matte.