Multi-Layered Keys and Rotoscoping for Green Screen

Even the best green screen footage can present issues that can't be corrected with standard keying procedures. Whether it's a particular movement in the scene or rough lighting, sometimes you'll need to use alternative methods to isolate your subject from the background.
In this segment we demonstrate multi-layered keys, and how to use masks to rotoscope your subject.
These advanced methods and can help you get the toughest green screen footage looking good.

Nobody wants to spend endless hours rotoscoping footage frame by frame, but sometimes, in order to get tough footage to work for your project, you'll find yourself facing that exact task. There are a few different methods for taming a tough key, and using a multi-layered key approach can be effective. Let's take a look at an example in after effects.
This full length green screen shot has two distinctly different shades of green. If I turn on our keylight effect, you'll see that the back wall keyed out nicely, but the floor area still needs some work. I'll select our foreground layer, and create a duplicate layer by pressing control d or command d on the keyboard. Now I'll use the rectangle tool, and draw a fairly tight mask around the the portion of our shot that still needs to be keyed out, making sure to include a bit of the well keyed portion in our mask area. Now I'll select the original layer, and draw a mask around the area of the shot that has a clean key, making sure there's some overlap between the two masks. Next, I'll select the problem layer, and reset the keying parameters by clicking reset in the effect controls window. Now I'll follow the normal process for pulling a key, which we've demonstrated in previous segments. This will allow me to have separate keying settings for the two areas of my shot. Simply dividing the areas in the shot can help address the issue with good results. You can use as many layers as you need in order to get a a clean key. You can see there's still some issues around the foot area, and I can select the pen tool to draw a mask around the area I need to eliminate... then select subtract from the mask dropdown menu. And now this key is looking good.
Rotoscoping is the process of manually creating a matte from your footage, in order to composite it over a background. When you've got tough green screen footage, you can use keyframed mattes or masks to achieve this. Let's take a look at the same clip in after effects.
Before I start drawing masks around my subject, I'll review the footage to make sure I know how many masks I'll need to isolate my footage. The best approach is to break up your subject into different sections, such as the head, arms, torso, and legs, and mask each section individually. In addition, you can create subtraction masks for the areas within your subject that need to be eliminated, such as the gaps between the legs and arms. It's a good idea to go into the edit menu, and select preferences, then appearance, and be sure that cycle mask color is checked. This will automatically give each mask you create a unique color to prevent confusion. Pick the point in your footage that has the most complex outlines, in order to ensure that additional mask points wont be necessary, then work forward and backward from that point to keyframe your masks.
I'm going to start on this frame, and select the pen tool. I'll zoom in on my subject by using the scroll wheel on the mouse. You can also use the dropdown box. Now I'll draw a tight mask around the actors head... each click on my mouse adds a new point, and clicking and holding will create a bezier curve. Try to use the minimum amount of points you can, while still maintaining a concise outline of your subject. It's okay if your points aren't exact, but try to make sure you add all the points you need. I'll close out the mask by clicking on the original point. Now I can refine the mask by adjusting individual points. To change a point back and forth from a bezier curve to a hard corner, hold control alt, or command alt on the keyboard. You can also change the mode of your mask to none if you want to see your background. I'll make some final adjustments until I've got a nice mask around the head. If I switch the mode back to add, you can see that the mask is a little too crisp. I can drop down the mask parameters to make some adjustments. First, I'll adjust the mask expansion inward by 2 or 3 pixels, then I'll boost the mask feather by 2 or 3 pixels. This gives the mask a more gradual edge, and eliminates the green outline. I'll change my mask mode back to none... and now I can begin animating this mask over time.
I'll click the stopwatch next to mask path to create a keyframe for my mask shape. And then I'll move forward frame by frame to check for changes in motion. Whenever possible, it's best to select group of keyframes to move in concert to prevent your mask from changing too much. You can select multiple frames by using the selection tool, clicking on a line in the mask, then clicking and dragging a box around the points you want to move. Holding shift will allow you to add more points to your selection. You don't have to animate every frame if there's no motion, but be sure to check each frame for accuracy if you skip them. From here, it's a matter of keyframing the head for the length of the clip. Once I've got the section tracked well, I'll go ahead and lock the mask, to prevent modifying it accidentally. I'll skip forward in time a bit, and show you how all of the masks look as I scroll through the timeline. The masks for these holes in our subject have been keyframed, and set to subtract to get rid of the background. If you need to turn a mask off at a certain point, you can simply set a keyframe for the mask path, move forward a frame, click shape, and enter 0 in each field. You can also move backward a keyframe, and use the same process to turn a mask on. You can see the sheer number of keyframes I've used in order to get our subject isolated. I'll turn on our white infinity background. And play back our clip to see the results. While it took some time, this looks pretty good.
Pulling a good key takes lot's of practice. When you combine solid keying skills, multiple layer keys and rotoscoping, you can tackle even the most difficult footage and make it work. So don't be afraid to break out every tool you have, and use them together to solve your issues, and take your green screen work to a professional level.