The Horrors of Rotoscoping

In the world of video production, there are some horrifying things we face regularly. Ok, ok, it’s not quite that dramatic, but rotoscoping has been long respected as a time-consuming and difficult art form, but hated, considered at best a necessary evil. 

So What is Rotoscoping?

By definition, rotoscoping is a technique where a subject — either live or animated — is traced over, frame by frame, to create a matte so that it may be composited with a different background or environment.

The term “rotoscoping” comes from the name of a piece of equipment which could project one frame of live action film at a time, originally with an easel containing frosted glass, upon which an animator could trace the live subject as a drawing using paper on top of the glass. The device was called a rotoscope.

While this device was used for many years, modern rotoscoping takes place on computers pretty much exclusively now. Programs such as Adobe After Effects, Imagineer’s Mocha and Silhouette all have powerful tools to help make the roto artist’s job easier. Primarily, a roto artist will use masks to create their mattes, often using multiple masks to assemble the shape that needs rotoscoping. 

Rotoscoping? Since When?

The rotoscope was invented by Max Fleischer around 1914, and he used it to create a three part series called “Out of the Inkwell.” The series was created specifically to show off the potential of the rotoscope.

As for talent, Max used his brother Dave to portray the series’ character, Koko the Clown. As Dave was performing as a clown on Coney Island at the time, he was a natural fit, and inspired the look and live action movement reference for the groundbreaking show.

Dave would perform on film, and the rotoscope would project his image onto Max’s easel, so that Max could trace his movements to paper, one frame at a time.

Rotoscoping has been long respected as a time-consuming and difficult art form.

In 1915, Max patented his invention, and before long it was being put to use in productions from “Betty Boop” to “Snow White and the Seven Dwarves.”

Over the years rotoscoping has been in constant use, not only on television and in movies, but also in music videos. Famous examples are The Beatles’ “Lucy in the Sky With Diamonds” video and A-Ha’s “Take on Me.”

Screen grab of A-Ha's music video Take On Me
Screen grab of A-Ha’s music video Take On Me
The shaky lines in “Take on Me” are the result of what is called “boil” or “jitter,” which is caused when an image isn’t traced consistently from frame to frame. The boiling in A-Ha’s video was intentional, but careless rotoscoping can lead to this happening without intent.

To give an idea of the amount of work that goes into a roto-heavy production, the “Take on Me” video involved rotoscoping 3,000 frames of live action video, and it took more than 16 weeks.

More famously, rotoscoping has been used in all of the “Star Wars” movies to create their iconic lightsabers. To create these a matte was manually added to each frame of the stick held by each of the actors, then a line with a colored glow was added.

Seems like a lot of work, but the results were fabulous! 

Sounds like fun. Why do we all hate it?

Rotoscoping has gotten a bad rap over the years, and for some good reason. Rotoscoping is a time-intensive process, laboring over every frame that needs compositing. When shooting video at 24 or 30 frames per second, those frames add up in a hurry.

For special effects, it’s tough to criticize the process, but there are many occasions where rotoscoping is necessary as a result of sloppy production work and, ultimately, a decision to “fix it in post.” 

Goulish Tales of Roto Horror

There are many examples of times when rotoscoping shouldn’t be necessary, or isn’t properly prepared for, but still ends up on the roto artist’s desk. Some common issues come up, even in tightly controlled studio environments. 

The errant gesture

During a simple green screen shoot, the talent reaches outside of the screen coverage. Regardless of the insistence of the Director or DP, the talent will always have a mind of their own and may not be interested in sticking around for multiple takes. Some talent will choose to ignore directions such as, “do not raise your hands over your head or they will extend beyond the top of the green screen,” or, “only walk as far as the tape marks or we can’t use the shot.”

The errant gesture is when talent being shot on a green screen allows a limb to pass outside of the area of the screen. In these cases, it’s possible to simply rotoscope out these few frames manually and add their mask to the keyed footage. 

The sloppy keyer

From wrinkled backgrounds to bad lighting, there are many factors that can mess with pulling a clean key and will necessitate some rotoscoping.

We’ve all been guilty of it at one time or another — not having the exact right tools to pull off good keying. Forgetting clamps to pull a screen tight, not having sufficient light to ensure even light across the entire screen or having a cable run through a shot are all possibilities when we’re not careful.

Regardless of which of these gaffes are made, there is a chance that the fix is to rotoscope the footage in post.

Screen grab showing sloppy key.
This is with the warning that there are great differences in the mattes generated by a keying plug-in versus rotoscoping. The edges created by rotoscoping are very clearly defined (though blur can be added if appropriate), where edges created by keying are based on which pixels fit the profile of the screen as defined in the plugin settings. Combining the two methods is an art form in itself. 

Replacing the Inexperienced Roto Artist

When rotoscoping something, such as a person walking, it is common for inexperienced artists to start at a random frame and create their matte in one large mask selection, surrounding the entire head and body. While this would be fine if the person wasn’t going to move, this is impractical for managing issues like when an arm or leg passes behind another part of the body, or as body parts bend.

While keeping the basic shape of the subject is possible, the number of points required to make the selection will likely vary, meaning there will be too many points in some places and too few in others.

The solution is to look at the body and think of places where basic shapes can form a portion of their outline while facilitating the type of movement going on. Multiple masks are necessary to really free up the roto job — create a shape for a knee joint, or for an ankle joint, then create a mask for the lower leg, upper leg, feet, hands, lower arm, upper arm, torso, head and pelvis. This way the shape for the calf and foot can remain relatively constant while the joint requires little change from frame to frame as well.

A good tip for creating these masks is to make them each on their own solid layer, so that they can be toggled on and off without affecting the video layer. 

The Director who doesn’t know what he or she wants

Sometimes a roto artist will receive a section of footage and be asked to remove an element — say a background — from the scene. If the footage is thirty seconds long, shot at 24 frames per second, that makes 720 frames requiring the change. In actuality, the final cut of the scene may only use frames 35 through 100. The solution? Demand direction of which portion of the clip requires the rotoscoping done. 

Is manual rotoscoping the only way to do it?

A recent development in the world of rotoscoping is Rotobrush. Rotobrush is a tool in Adobe After Effects (CS5 and later) which allows an artist to make a selection (similar to the Quick Select tool in Photoshop) and by tracking the footage forward or back the selection will — if everything goes to plan — follow the selection.

While not as foolproof as rotoscoping manually, this tool can be handy in a number of situations and save the artist scads of time.

Picture this — a client has an idea of how they want their headshot video taken. She asks to be shot in front of a wall in her office. No problem. The video team lights her up, gets her mic’d up and they shoot their video. Only problem? The color of the wall matches her skin tone too closely. Or maybe it clashes with what she’s wearing. Luckily, with the back being a nice, constant color Rotobrush can be used to select the speaker over the course of the clip and allow a new background to be composited into the scene.

Another fun thing to use Rotobrush for is creating perspective in a shot. Similar to cutting a Photoshop PSD into multiple layers and spreading it out in Z space in After Effects, Rotobrush can select a subject which can be duplicated and pulled forward in Z space. This is a quick and easy way to create that cool 3D perspective. 

How can I hate rotoscoping less?

Screen grab from Scanner Darkley.
There are lots of little things that will help new roto artists get more comfortable with rotoscoping. While no one tip or trick will fix every problem, there are some best practices that will help them keep their cool over longer projects.


  1. Choose a mask color that is visible. Like, really visible. Why struggle to see a mask path in an operation based solely around a mask path?
  2. Turn off the mask mode. Set it to “none,” versus “add” or “subtract.” Why peek through a mask to see what’s being worked on?
  3. Practice working with the pen tool. Get used to creating smooth bezier curves to cut down the number of mask points necessary. Remember when creating a smooth curve with the handle on a bezier curve point, it’s possible to adjust one side of the handle at a time.
  4. Look for negative space in the image. If it makes more sense, strip away what isn’t necessary instead of trying to simply select what is.
  5. Break down large and complex subjects. Turn a person into a bunch of selections which combine to make a great matte.
  6. Use separate layers for each mask and label each layer clearly.

There are many more tips, and many experts out there that can impart more knowledge about rotoscoping. It is a set of techniques, tricks and methodologies which has evolved over decades to what has become modern rotoscoping.

The best tip of all is to look at how as many people as possible handle their rotoscoping projects and take away the practices which work best for you. 


Steps to a First Rotoscoping Effort

When it’s time to get started rotoscoping, try some test runs. Take little pieces of stock footage with a prominent subject and work on removing the subject or background consistently.

The first step in any roto job is to assess the shot. There are many ways to accomplish the same goal, so decide your plan of attack.

Create a mask around the subject (or multiple masks if it is a complex shape, such as a person). Keyframe the mask path at the first frame of the shot.

Track the footage with Mocha, After Effects motion tracker, or whatever tracking tool is available. Even just keyframe the mask position in After Effects roughly over the course of the composition to provide a starting place.

Add the tracking data to the solid layer(s) that the mask(s) sit on. Having the subject tracked will minimize the amount the masks will need to be adjusted in each frame.

Instead of keyframing every frame, scrub to the final frame of the shot and adjust the mask(s) to fit the shape at the end.

Drop back to around the halfway point and look at the mask(s). The mask will need adjusting. Adjust the mask to fit the shape.

Continue adjusting in this manner — check the mask path at halfway points between keyframes and adjust. Repeat until the track is accurate.

This can be a time consuming process, but the results of rotoscoping can be fabulous. The more time and effort that goes into a roto job, the finer the results can be. It can be a truly rewarding process!

Russ Fairley owns a Toronto-based video production company and is the host of RFShow.TV.

Related Content