Researchers at Netflix have published a paper on a revolutionary new system for chroma keying. The Magenta Green Screen technique uses a regular green screen background but lights the actors with only red and blue foreground lighting.
What is chroma keying?
Chroma key is the visual effects technique where the actors are filmed in front of a green screen. The green background is then made transparent in post-production so that a new background can be composited behind the actors. The transparent part of the image is called an alpha channel. Chroma keying is also called green screening because of the color of the background. However, when the technique was first developed the background screen color used was blue. Blue screens are also still used where the actors clothing or other foreground elements contain a lot of green.
Why is Netflix using artificial intelligence (AI)?
There are a lot of software solutions that you can use to remove a green screen background and replace it with the image you want. Most of the time, these work well; however, there are some challenges. For example, fine details such as hair can be difficult to key accurately. There are also issues with transparent objects, items that are the same color as the background, and motion blur around moving subjects. These issues need to be touched up on a frame-by-frame basis. This is why you see so many visual effects artists listed in the credits of films and TV shows.
What is magenta green screen?
The Netflix researchers are attempting to develop a system that will automatically give more accurate keying of the subjects. With magenta green screen, the background is green as usual. However, the actors are only lit using red and blue lights. This means that there are no green elements in the subject part of the image, so the background can be keyed out easily. Unfortunately, this still leaves the subjects looking bright magenta and very unnatural.
Machine learning
To color correct the video once the alpha channel transparency had been created, the researchers first simply added the green channel back into the footage. However, in the resulting images, the subjects were yellow and washed out which was still unusable. The researchers, therefore, trained a deep neural network, or AI, to understand more accurately how to replace the green elements of the video and restore a natural-looking image. The images in the research paper from Netflix show that the AI-corrected images look very good.
What next?
Although the correct colors can be restored with the use of AI, the poor actors still have to work in the unnatural magenta-lit environment. The researchers, therefore, experimented with a technique called time-multiplexing. Put simply, this means switching the magenta light on and off very quickly so that the normal lighting appears constant to the actors. However, flickering lights can be dangerous for a person with a sensitivity to flashing light. In addition, the camera has to be synchronized with the lighting so that it only captures an image when the magenta light is on. The research paper concludes with recommendations as to which direction the research should head next.
What we think
The majority of films and TV shows use green screen or blue screen visual effects to some extent. However, a lot of the post-production work has to be done by human artists. This is the reason why visual effects make up such a big part of many production budgets. If this work could be automated, then costs would be reduced and the post-production times would come down as well. The magenta green screen technique from Netflix is still very much at the experimental stage but it could be one solution for the future.
