Fixing Rough and Discolored Edges for Green Screen

Green Screen work is complex, even under ideal circumstances, and working with footage that hasn't been lit or shot properly can be pretty tough. In this segment, we show you how to fix those rough and discolored edges that are often left after pulling an initial key.

Video Transcript

Green Screen work is complex, even under ideal circumstances, and working with footage that hasn't been shot properly can be pretty tough. In this segment, we talk about fixing those rough and discolored edges that are often left after pulling an initial key.
Using chroma key in premiere pro, ultra key in premiere pro, and keylight in after effects, we'll fine tune those edges to get a great looking key.

When you see the sheer amount of work that goes into big-budget film green screen work, it's no wonder that sometimes our own results may not match up to our expectations. After doing the initial work to pull a key, the subject is often left with a green outline, or jagged edges.
One reason this can occur is due to bad lighting. If the subject is placed too close to the wall, or a rim light isn't used, it can create some difficult issues in post.
Chroma sub-sampling of your footage can also contribute to the issue. The pixels that make up your footage have information about Luma, or brightness, and chroma, or color.
If you remove the chroma from your pixels, you end up with a greyscale image. However, if you remove the luminance from your pixels, the picture is black. Viewers are much more sensitive to luminance accuracy than chroma accuracy.
In order to try to save space when recording images. Engineers developed a way for multiple pixels in an image to share the same chroma data, with acceptable loss in perceived quality. The number of pixels that share the same data determines what the subsample rate is. Let's take a look at 3 different types.
The common way of writing out the chroma subsampling ratio is J, a, b. 'J" represents how many pixels wide the sampling pattern is. Typically it's 4 pixels wide. The "a" number represents the number of pixels in the top row that will be sampled for chroma. The "b" number represents the number of pixels in the bottom row that will be sampled. In this example, each pixel gets sampled for Chroma information, resulting in 4:4:4, which means no chroma sub-sampling is being done. Typically, only expensive high end cameras record this way. 4:2:2 subsampling has two pixels on the top row that share chroma and two pixels on the bottom row. Essentially, this is only 50 percent of the chroma information of the 4:4:4 sampling rate. Formats such as Panasonic DVCPROHD, and SONY XDCAM HD 4-2-2 use this method. A third common ration is 4:2:0. This ratio takes two samples from the top row, and none from the bottom, which means the bottom row is sharing the chroma information for the top row. Essentially this is only 25 percent of the the chroma information of the 4:4:4 method. This ratio is commonly used in HDV, and MPEG formats including most DSLR's on the market today. Finally, we have 4:1:1. this ratio takes one sample from row a, and one sample from row b, again returning only 25 percent of the 4:4:4 method, and is most commonly found in the DV and DVCPRO formats. Chroma Subsampling is a big issue when dealing with green screens, because the color detail that is lost degrades the edges between your subject and the screen.
Luckily, there are some tools we can use to help counteract the shortcomings of our footage. Let's start by demonstrating chroma key in premiere pro.
This is a very basic keying tool, and we pulled the initial key in our previous segment. You can see that we're left with this distinct green outline. Looking at our effects window, we have a couple different options to try to fix this. Making some major adjustments to the blend will start to get rid of the green, but if we check the mask only box, we can see that this causes us to lose some key opaque areas. Ultimately, this effect on it's own isn't sufficient to get rid of the edges. Using a 3rd party plugin like the FILL: choke effect from RE:vision Effects can be effective. Applying this effect, and increasing the choke amount sparingly addresses the issue.
The good news is most current editing programs, including premiere pro cs5 have keying effects with some greater flexibility. Let's look at the same shot using the Ultra Key effect.
We pulled the initial key in our previous segment, and while this looks pretty good, we do still have some edge issues. We'll go into the Matte cleanup section of the effect to make some adjustments. In this case, due to the poor lighting, we'll use just enough of the choke parameter to reduce the green edge. In order to smooth out the edges and have them blend a little better, we can increase the soften parameter. As always, adjust the least amount possible to get the desired effect. We'll view it in motion to check for any issues. And this looks pretty good.
Keylight in after effects is a powerful keying tool, and we'll use it on the same footage.
We pulled the intial key in our previous segment, and as you can see, there actually isn't a whole lot of green present. There's still a bit of an outline though. The reason the outline isn't green is that keylight is already suppressing the green on those edges for us. Switching the view to intermediate result will show you the edges before color correction. Let's switch back to final result to view the edges after color correction. The shrink grow parameter can be increased or decreased to expand or close in your alpha matte. Use the smallest adjustment possible to get rid of those pesky outlines, but watch your edges closely to ensure you're not noticeably eliminating part of your subject. You can see this did eliminate the bulk of the issue. If your shot seems a bit too crisp and isn't blending the way you want into the background, you can adjust the screen pre-blur to get a softer edge without re-introducing edge issues. We'll take a look at this in motion... and this looks pretty good.
It takes a lot of effort and adjustment in order to get good results from challenging green screen footage. Getting a good key with clean edges is only part of the process. In our next segment, we talk about removing unwanted green spill from your footage. It's one more green screen issue that you can fix, to get your footage looking good.