Next3D's plan to bring recorded video to the Oculus Rift

The dream of wearing a lightweight headset, like the Oculus Rift, in order to simulate physical presence isn't limited to the imaginary worlds of video games. One man's vision is that of immersive TV shows, movies and live sports. In fact, David Cole, co-founder of Next3D and an industry veteran who helps content creators and providers produce and deliver 3D, has been using his Rift dev kit to bring TV and film to life since the kits started shipping in March. The company is combining its video processing and compression technology with its experience in content production and stereoscopic delivery to offer what it's called Full-Court.

Next3D hopes to leverage its existing relationships with creators and providers to assist them in jumping into the world of live-action VR content. This includes both pre-recorded and live broadcasts. We wanted to see this firsthand, so we jumped at the opportunity to witness the creation of content and experience the results. This trial run of Next3D's stereoscopic, 180-degree field-of-view camera rig, and the post-processing to adapt it to VR, was part of the production of the paranormal investigation show, Anomaly, at Castle Warden in St. Augustine, Fla. Being nearby, we braved the perils of the haunted surroundings to tell you about what we hope is only the beginning of virtual reality content.

How it works

The name Full-Court certainly fits Next3D's vision of virtual reality. We can't think of a better way to quickly describe its intention to create the feeling of truly being there. This immersive visual and audio content is captured using a rig with dual Red 4K cameras fitted with wide-angle fisheye lenses and 3Dio binaural microphones. The twin 180-degree field-of-view lenses allow the 3D rig to capture everything you might be able to see from the same vantage point. This makes it possible to leverage the head-tracking abilities of a VR headset to allow the viewer to look anywhere in the field of view. The plethora of pixels in the 4K cameras far exceeds the resolution of the Oculus Rift, but they're essential in order to fill every pixel of the headset, no matter where the viewer is looking.

In order to capture the experience of attending a sporting event, Next3D would set up its rig in a static position at center court (as an example). The single camera position would capture anything you could see from that point, without the need to ever pan or tilt. The 180-degree range would allow the viewer to move their head to follow the action, just like they would have to do when attending a real game. Multiple camera positions could still be leveraged. In fact, Cole tells us he envisions an interactive way for the viewer to choose the perspective -- giving the user the ability to jump in and out of different angles, while limiting the jarring feeling of being transported without their control.

The fisheye lenses and 4K cameras are only part of the process, though. Next3D works its real magic in post -- its current business consists of real-time processing of 3D broadcasts. The system, which relies on a special encoder and decoder, takes the raw footage from the two cameras and delivers it to a virtual reality headset. In its current iteration, this takes place on a computer via a QuickTime plugin and playback software running on a PC with an Oculus Rift development kit connected to it, but who knows what it might look like in the future.

The raw images captured from the 3D camera rig.

The images after post-processing, ready to be rendered on the VR headset (actual image zoomed at request of Next3D).

Live-action VR content

The company doesn't see sporting events as the only application, either. Next3D's dream of virtual reality content caught the attention of veteran documentary filmmaker Michael Watchulonis, founder of 3DigitalVision. He's currently the executive producer and director of the paranormal investigation series, Anomaly. Eager to be the first to try out the new format, his crew visited their favorite documented paranormal site; Castle Warden in St. Augustine, Fla.

Ever since the 1880s-era building became the Ripley's Believe It or Not! Museum nearly 70 years ago, employees have reported paranormal activity. Waiting anxiously until after the final ghost tour of the night completed, Jack Kassewitz lead the investigation with the 3D rig ready to capture the entire experience. He followed two museum employees who shared their firsthand experiences with ghosts as he tried to document, measure and explain what appeared to be connections to the spirit world. In addition to the Red 4K rig, Next3D also set up an infrared 1080p 3D camera. Even though the resolution is less than 4K, the IR illumination results in a better-lit experience -- the released episodes of Anomaly use 3D infrared cameras extensively.

Observing the production left little doubt that there's a lot to master in capturing recorded content for virtual reality. Framing seemed particularly challenging as moving the camera mid-shot isn't an option. Anything but a simple dolly forward or back is out of the question as it's too disorienting for the viewer, according to Cole. Add in the requirement that the camera must stay level -- the head-tracking assumes the viewer starts while facing forward with their head level -- and the director has a lot of limitations to work with. None of this discouraged Watchulonis, though. He was excited to be the first to figure out the new format. He even shared with us his desire to work out all the kinks in time to have content ready when the Oculus Rift finally reaches consumers. Both Watchulonis and Cole share the dream of immersive content and neither needed a focus group or corporate committee to tell them this was a movement they wanted to be a part of.

Our experience

The end result would give us a taste of the future of home entertainment. Despite the lower resolution of the development kit, and the visual artifacts that go with it (screen door effect, motion blur and scaling / softening), the immersive feeling of the Oculus Rift, with headphones, is something you want to experience. It starts by placing the surprisingly lightweight headset on, while seated. Next comes a nice set of headphones for piping in the surround sound. At this point, you are surrounded by darkness and the absence of noise, but that ends quickly as your eyes adjust and the image appears. It is hard to explain, but it is more like looking at the real world than watching a TV or computer display.

We had the chance to experience three different realities; a pickup basketball game, a day of power paragliding at the beach and the investigation from Anomaly. In all three cases, the camera captured the moment from a single perspective, but the headset tracked where we wanted to look. Assisted by the surround sound, this was one of the most immersive live-action experiences we've ever had. It's very much like wearing a mask, though, which means limited peripheral vision. And while the 3D imagery and surround sound did provide an engrossing feeling; the visual artifacts, narrow perspective and inability to change vantage points by moving around kept you grounded in reality. Full-Court did succeed at allowing us to relive our visit to Castle Warden.

Instead of being at the mercy of the camera operator to catch the disturbances occurring in the dark, we were able to look throughout the room ourselves to wonder about every shadow. We couldn't actually move around the area or explore other rooms, though. The fact that the rig's 180-degree range captures everything in the space can cause problems if you try to use multiple cameras, as the viewer can see those other setups. For sports and documentaries, this is less of an issue, but this could really make scripted virtual reality, live-action content difficult. We imagine it'd be possible to instantly jump into any room or perhaps a walkthrough of the hallways and staircases, but these are the early stages. Hopefully the techniques to facilitate those shots will be mastered soon.

Next steps

From here, Next3D will continue to work with its partners to figure out how to use the medium for live-action content, helping set up the camera rigs and performing the post-processing on the raw video to adapt it to the Oculus Rift VR headset. While Cole confirmed there were a number of content partners interested in testing out the technology, only Watchulonis of Anomaly was ready to announce a partnership with Next3D. It's doubtful they are the only ones working to develop this market, but like the rest of us, everyone is likely waiting on the Oculus Rift to make it to consumers.

Wrap-up

While our demo time was far from a full feature-length film or an entire live sporting event, the moments we did spend with the headgear on were convincing. Current 3D content leaves a lot to be desired and doesn't come close to the immersive feeling of virtual reality, but what we experienced was far different from the square box that has been serving us well for years and producers and directors have a lot to learn -- a whole lot. It's wondrous to watch the excitement of film creators as they embark on a totally new format, but at the same time, it's obvious that this is a whole new world with little resemblance to the one we know now. We wait anxiously to see what might come of the current renaissance of virtual reality and if Cole and Watchulonis have anything to do with it, it won't just be for those who enjoy video games.