As society has relied more and more on video calls and conferencing, high bandwidth is a problem for many. However, NVIDIA Research has invented an AI that can dramatically reduce video call bandwidth and improve the overall image quality.
Essentially, the AI can reduce the required bandwidth for video calls by swapping out traditional h.264 video codec with a neural network. PetaPixel gave an example of how this would translate into practice. “the required data rate fell from 97.28 KB/frame to a measly 0.1165 KB/frame – a reduction to 0.1% of required bandwidth,” PetaPixel wrote.
How does the NVIDIA AI work?
As we said above, the NVIDIA AI replaces traditional full video frames with neural data. Video calls usually send out h.264 encoded frames — which can be quite taxing due to high amounts of data. Previously, pixel-packed images were sent. Now, the AI will send specific reference points on the image around the eyes, nose, and mouth. Once that reference image is sent, the receiver’s generative adversarial network (a type of neural network) uses the reference with the key points to reconstruct the image. Overall, this process sends a substantially lower amount of data. The reason why is because the key points sent are much smaller than the full pixel images. So now, slower internets can display a clearer image with less buffering.
Since the neutral network is using reference data to construct its image, the NVIDIA AI can construct a new camera angle. What do we mean by this? The reference data can be used to make it seem like the subject is looking directly at the screen when they’re not. This feature is being called “Free View.” The feature allows people to use a separate camera off-screen and still appear to be keeping eye contact with the camera.
How will this technology be used in the future?
Currently, there’s no information on how this technology will be implemented. We can see it really helping virtual workflows. However, the technology could make deep fakes much more believable and harder to pick out as fake. This could pose a huge problem for society. You can read more about deep fakes and how it poses a threat to our society here.
Image courtesy: NIVIDIA