For years virtual reality technology has remained intractably saddled with Lawnmower Man-era content. Even with the advent of affordable, consumer-targeted VR systems like the Oculus Rift, the most common application has been immersive video games. That's all about to change.
What if we could generate not just photorealistic, but actual live-action content that not designed to transport you to some virtual CGI universe but rather anywhere on Earth, dropping you in the middle of any event? This sort of experience—more Strange Days than Stephen King—is closer than you'd think.
The Jaunt VR camera
Jaunt VR, a startup that has developed a radical new method of generating live-action virtual reality content, has teamed with New Deal Pictures to produce live-action content for virtual reality headsets. Jaunt leverages a custom-built camera rig with a proprietary stitching algorithm to generate, not just 3D, but fully immersive content.
Current cinema technologies — whether IMAX, IMAX 3D, or Smell-o-Vision — doesn't entirely immerse the viewer within the plot, no matter how realistic the on-screen action, it is still limited to a projected image on a flat screen. What's more, they still only offer passive interaction with the subject matter as opposed to the immersive first person perspective that gaming franchise titans like Halo or CoD can offer player. Jaunt and New Deal, though, want to be first to the next generation of movie-watching.
A New Deal PA attaches the Jaunt camera to a parachute rig which will be dropped from a height of nearly 50 feet to provide the viewer with a sense of dropping into enemy territory - [Image credit: Jaunt VR]
"Everything is different because everything is in the shot," Owen Remeika, a rep for Jaunt told Gizmodo on a recent overcast Sunday amid the cow pastures of Petaluma, California.
"Jaunt uses multiple lenses and sensors on their prototype camera to create immersive, stereographic, 360 degree VR video ("cinematic VR")," Jens Christensen, Jaunt VR CEO, said. "Unlike traditional 3D, there are no stereo pair cameras on the Jaunt camera. Instead, Jaunt's 3D is computed algorithmically."
[Image credit: Jaunt VR]
This is an important distinction, Scott Broock, VP of Content for Jaunt VR, explained as we stood on the set of The Mission, a roughly 10-minute-long live-action film following a trio of WWII American soldiers as they carry out a dangerous mission behind enemy lines. Rather than rely on the bulky pairs of stereoscopic cameras conventionally used to generate the 3D content we watch in cinemas, the Jaunt 360 degree camera combines the optical inputs of more than a dozen cameras to create a unified, nearly omnidirectional view of the scene. The only bit you don't see within the virtual environment is the area directly below your point of view, where your torso and legs would be. Instead, it's replaced with the Jaunt VR logo. The algorithm similarly stitches together audio from its binaural mic, which results in Dolby Atmos-like surround sound.
However, the new filming technique does still have some kinks to work out. Besides not (yet) allowing the user to move about within the virtual space on their own accord — that is, they can look around but not walk around — the production crew must wrestle with the task of flagging or otherwise directing a viewer's attention within the scene but not so obviously that it distracts from the plot. This is easy in traditional cinema because the only action is that on the screen in front of you. But when the user can look at anything in the virtual room — say, ringside at a boxing match — keeping their attention on the primary action (the fight) and not simply looking around (at the blonde in the fourth row) can be difficult.
[Image Credit: Gizmodo]
The actual production of these films also pose new challenges. Since the cameras pick up everything in a 360 degree ring (as well as above and below), there is nowhere to stand when filming where you won't be in the shot.
Matt Gratzner (L) directing - [Image credit: Jaunt VR]
"What's exciting about it, I feel like we're in 1927 when sound was first being introduced," The Mission's director, Matthew Gratzner said, "but what you have to realise is that you don't really have the ability to do coverage. In other words, you're looking 360 degrees so every time you're shooting something you have to be aware of someone or something you could see in the background [that shouldn't be there]. There is not 'behind the camera'."
Cast and crew peeking out from cover during a take
On the set of The Mission, this resulted in the entire cast and crew ducking into sheds and hiding behind obstacles when the cameras were rolling.
What's more, the director must shoot blind, so to speak, because the software algorithm that generates the VR content requires 15 seconds to compute every second of action. This lag means that the director can't call for a cut and then immediately see if the shot worked on a reference monitor; he has to wait 15 times as long as the scene took to shoot to see if it actually came out OK.
These procedural challenges aside, the potential of this cinematic technique (and the Jaunt camera system that enables it) is nearly limitless. Jaunt demo'd a few clips for us last Sunday, including sitting me ringside at a title fight (where I missed the knockout punch because I was ogling the crowd) as well as standing me both behind the DJ at an enormous outdoor electronica concert and front row, centere in the crowd. The experience is difficult to describe, though I did find myself repeatedly saying, "this is the coolest thing ever," during the demo.
[Image Credit: Gizmodo]
And the system is, clearly, not limited to non-fiction. The Mission is just the first crack New Deal Pictures and Jaunt are taking at VR storytelling. The studio is also considering horror and sci-fi tales for future projects, which, if The Mission is any indication, will be more real and more terrifying than anything you've seen before.