These scenes are not part of the latest video game. This is news gathered for a new medium. The video can be found on the New York Times web site or in their new app NYTVR. The 11 minute story called “The Fight for Falluja”, has gained a lot of attention. Partially by bringing spherical video images from an intense region into your goggles, and also because it demonstrates great journalism done in a new way.
It would seem that not just the New York Times has jumped into the VR wave. The NBA committed to bringing us court side to a game in spherical video each week of the 2016-17 season. Conan O’Brian has done a show in spherical video. Hollywood producers are creating original dramas in spherical video and offer VR glimpses of new movies.
Does this signal a shift in video? Is this the next jump in production like moving from standard definition to HD? We spoke to a variety of experts in spherical video to find our what’s being done and what this shift looks like. The overall consensus is that this is certainly something new but it’s too early to tell if this will have the same impact. VR or Virtual Reality has been on the horizon for some time but suddenly technology has given way to an explosion of content; it has certainly gone beyond a gimmick.
The Way Forward
First a little terminology is in order. There are Spherical videos created using multiple cameras. The images are brought together in a process called stitching. Videos are found online or in an app that people watch on a variety of VR goggles. The simplest, and lowest quality viewing experience, is a cardboard viewer referred to as Google Cardbaord. Google released the simple fold up viewer in 2014. There are new concepts being introduced called AR or Augmented Reality. This is a experience where you view in 360 but things are added to your view that you can interact with. A recent example of this is Pokemon Go, a mobile game.
Jeremy Bailenson is the founding director of Stanford University’s Virtual Human Interaction Lab and the Thomas More Storke, Professor in the Department of Communication. He studied the physiological and psychological effects on VR and human behavior for decades.
Professor Bailenson tells us that the spherical video concept has been around for some time. “Modern VR technology really began as conversation between technologists and science fiction writers. The head-mounted displays and virtual reality software that we see today have been developing over the past several decades.” You can see evidence of this in some classic sci-fi movies and television show. Think the holodeck in Star Trek the Next Generation.
The development of immersive technology has been in the works much longer than many might think. “It's fair to say that VR began back in the 1920s, with the first Link flight simulation (1929) — humans have been striving to create realistic feeling virtual experiences for a long time. The VR of today does seem to be here to stay,” says Professor Storke.
There are examples of the technology jumping from military applications to entertainment as early as the 1950’s. Disney developed a system of 9 film cameras that would be synced to 9 projectors in an technique called Circle-Vision 360.
It’s Bailenson’s opinion that we are not seeing the next jump in the industry but something entirely new, “I don't see every type of media content working for VR. However, for the home-run applications, VR will be immensely powerful and will augment our everyday lives in ways never seen before.” He says that one limitation is the ability for directors and producers to “lead” the audience in which way to look. Because the experience is all around, spherical stories need to create a way to focus the audience to look in direction that the producers want you to look.
The Producers’ Art
Graham Roberts is the Senior Editor, Graphics at the New York Times and one of the pioneers of their spherical video news gathering. He says that the Times views this as extension of the visual journalism. They produce a video each day and call it the “Daily 360”. He says, “I don't think it's a replacement or anything. I think it's just another tool. What differentiates it is sense of presence. It’s is more like photography. It’s just like a moment captured, like a snapshot.”
He says that in this case the direction the viewer looks may not even be an issue. The New York Times produced a piece during the 2016 Presidential Campaign called “The Contenders.” They brought a spherical video camera to a variety of events with different candidates. Graham says that it's an experience that lets you look in many different direction and each time you get a different perspective on the event. He refers to this as that “sense of presence.”
Graham says that for the industry this is actually a new way of communicating. “We need to understand a new language of storytelling which hasn’t been invented yet. Not just for the creators but for consumers. With film there's kind of a language with shots and editing that everybody understands. It’s not really there for VR yet.”
Hollywood is working on this new language. One of the first scripted spherical series is called “The Invisibles.“ It’s produced by 30 Ninjas Productions, headed by Hollywood heavyweight Doug Liman. His list of credits includes all of the Jason Bourne movies. We spoke to the program's Executive Producer Julina Tatlock and Production Director Lewis Smithingham. They tell us that the series, which centers on a wealthy family who have the power of invisibility, was a natural for spherical production because of that sense of presence.
Julina tells us that, from a production standpoint, they were innovating the story and developing the “language” alongside the innovation of the technology. “We shot for a short burst of time. We shut down production, looked at the footage, edited the footage and evaluated, very honestly, what worked and what didn't work. We then wrote forward in the story, adjusting what we’ve already written.” The entire process took 9 months to produce the six part series.
We shut down production, looked at the footage, edited the footage and evaluated, very honestly, what worked and what didn't work.
Julina continues by telling us that the production was unlike anything else they’ve ever tackled. She says that they could not think in terms of the traditional movie set because the viewer would see everything. “We shot mostly on location so we would dress for all directions. We are ready for anybody to look in any direction.”
How does the crew stay out of the shot? She tells us, “You literally start the camera and hide.” They shot mostly on a Jaunt camera but they would include a smaller Ricoh Theta camera to their set up. The Theta will live stream to a smartphone so the director can still see in 360 but is a lower resolution. Crew members would hide under things and behind things and the scene was shot.
But how do you deal with things like lighting when the viewer sees every angle? Lewis tells us that they relied on award winning lighting designer John Ingram. “He worked very closely with the set designers to ensure that lighting was accounted for. There was a lot more lights and a lot more practical lights than we would have on a set.” They even created new lights for some of the shots using small LEDs.
For both the New York Times and 30 Ninjas Productions, the bottom line is the storytelling. Without a story, the spherical video is simply a gadget. Many companies are trying out new ways of spherical storytelling. Professor Bailenson adds, “VR is special because you can move your body and interact with your environment. You can certainly use VR to tell a story, but you have to consider what makes that story work with this type of medium.”
How It’s Done
So we have a great story to tell that might be great for a spherical video experience. How do we find a specific cameras and begin shooting in VR. It seems that the landscape changes weekly. Fortunately, gear is becoming more accessible and less expensive. Technically speaking, to achieve spherical video it requires multiple camera that are stitched together with software. You can simply get a bunch of cameras and set them up in a circle and rely on software. This can also be achieved in the form of a single unit with lenses all around that has the stitching built in.
GoPro has been right at the front of the current spherical movement. For several years, companies have made mounts specially designed for the small, go anywhere design of the HERO line. In addition, the camera comes equipped with a wide angle lens. GoPro lost no time in developing the OMNI, an aluminum cube that fits and controls six HERO cameras. Of course, action cam spherical video seems like a natural fit.
Alexandre Jenny is the Senior Director of Immersive Media Solutions at GoPro. He tells us “OMNI has the potential to help make the VR medium a reality. It is an amazing enabler for people who want to be involved in VR production, and I see it acting as an accelerator for getting high-quality VR content in-front of global audiences.”
He believes that this is the ultimate storytelling medium, “When you are immersed into high-quality VR content, the experience is very powerful. You feel as though you are in the scene. You fear height. You have empathy with the subject. No doubt, it is the best tool available to stimulate emotions and feelings.”
Guido Voltolina is the Head of Presence Capture with Nokia and he could not agree more, “We are at the very beginning and we’re rapidly emerging into not just a gimmick. There’s something more. If I tell a story in 360, it stimulates you into not just enjoying it but it happens all around you. Like in real life. It gives you a way to tell stories in a different way.”
Nokia made a splash at trade shows in 2016 with their pro level OZO camera. The device looks like an alien spacecraft and is fitted with 8 cameras and surrounded with mics to give spherical video and sound. Guido says that it’s being used in some exciting ways. Like a recent European soccer championship that was covered with 7 OZOs and it was live streamed and switched. They even mounted a camera under the cup when the cup was raised, “It doesn’t matter which player was coming from which direction. That point of view was unseen before. Even with 2D cameras.”
Guido believes that the emergence of VR is not a simple linear step,“It’s a different media. It’s like when you go from writing to picture and picture to video. 360 video is just a bigger screen.”
It's Just The Beginning
It is clear that there will be much more to come in the spherical video market. Is it the next jump in video production? It seems like it’s more of a lateral move.
Professor Baienson from Stanford explains, “VR will work wonderfully in many different contexts, but it's not suited for every type of media experience. There are plenty of media experiences that are already great as they are (cinematic movie experiences, for example). I don't see every type of media content working for VR.” He continues, “VR will augment our media-rich lives certainly, but it won't necessarily replace other media.”
Additionally, Professor Bailenson and others have pointed to the separation that wearing a headset causes. You won't be able to interact with others during the experience. However, there are some things on the horizon that may signal a change in that as well. For example, a patent was filed back in 2008 by USC for a 360 projector and as recently as November Samsung filed for a similar patent in South Korea.
Lewis Smithingham of 30 Ninjas reminds us that we are at the very edge of this, “ You should try to experiment, because right now this is such a new medium. The last thing we want to do is put constraints on it. Try, try try, is what I would recommend. Experiment and play and try to do something new and different. That’s how we will move forward.”
Some experts that we spoke to said that this time is like the very introduction of motion pictures. At first, people were in shock. They would see an image, like a train coming towards them, and jump out of the way even though it was simply a projection. Today VR is very much the same. So the question we’re left with is do we jump out of the way? Or do we jump on board?
SIDEBAR: Audience Participation
You’ve probably seen VR goggles of one type or another. They range from the simple cardboard or plastic units that you slip a cell phone into or the complex units like Sony’s PlayStation VR. Most have some type of channel or app that contains links to the VR material. Many of these are proprietary to certain viewers, so how is this media going to cross the boundary to delivery?
Currently, the most vibrant collection of open spherical media lies on YouTube. They launched their spherical channel in March 2015. A YouTube spokesperson told us, “We made a big, early bet on 360-degree and 3D video and continue to invest in immersive technologies. We will continue to push the boundaries of online video with new formats. We’re focused on bringing you the kind of immersive, interactive experiences that only VR can provide.”
For content creators, the upload process is pretty straight forward but it may require additional downloads. They have direct upload with some cameras like Ricoh Theta and the Kodak SP360. There is also an easy upload with Kolor Autopano, a stitching software. For other solutions you’ll need to go to the Google Support page and download the Metadata app. There is a Mac and a Windows version.
YouTube currently has the largest content library of spherical videos. You can find all different genre of media from action sports, too music videos, to travelscapes and other experimental concepts.
Jeff Chaves is the Chief Creative Officer of Grace Pictures Inc., which he co-owns with his wife, Peggy. He got his start as an Army Broadcaster in the 1980s and spent 12 plus years working on broadcasting. Jeff left broadcast television to pursue full-time ministry.