Legend3D’s VR Director Justin Denton on the charms and challenges of virtual reality.
Despite initial doubts, there is now no question that virtual reality is the next big thing. But, as an entirely new way of looking at the world, it brings with it a new set of challenges we hadn’t even thought of before – most notably, how do you tell a story when your audience is able to do whatever and go wherever they want?
“What’s most interesting is how quickly it’s evolving, that we’re starting to latch on to how you tell these stories,” explains Justin Denton, VR director & visual effects supervisor at Legend3D. “Time travels differently - it’s either slower or faster. We’ve gotten so accustomed to the editing pace of standard film, that now when you’re in a virtual space it’s very different.”
Denton himself is an expert of all things VR. He began his visual effects career in pre-visualization, before moving into the not unrelated field of supervision, then virtual camera development. Three-dimensional cinematic worlds obviously translate into virtual reality, but it was Denton’s later work on theme park and planetarium experiences that set him up for his role at Legend3D.
“The main person I learned under was Richard Taylor, who was the VFX supervisor from the original Tron,” says Denton. “He was dealing with all these things that people think of as new VR problems, but he had been solving them for decades. So I understood some of the inherent problems of telling a story for VR really early on. I spent two years there, and just this January I came on with Legend3D.”
Like Denton, Legend3D was already cut out for the burgeoning virtual reality industry. It made a name for itself as a post-production visual effects house, converting films such as MAN OF STEEL, TRANSFORMERS 3 and 4, and MALEFICENT from flat 2D to stereoscopic 3D. It felt it could use its well-established pipeline to solve one of the biggest problems with shooting live-action footage for virtual reality: the chunky, clunky cameras which capture the multiple angles for 3D playback.
Legend3D completed stitching and conversion on The Recruit: R U In to launch its virtual reality division. The company also provided some simple visual effects compositing for the live-action short film, which was made for Samsung’s Galaxy Gear VR headset. But its next project, for tequila company Patrón, took Legend3D further down the virtual reality rabbit hole.
This advertorial virtual reality experience was pitched, shot, and edited by Legend3D in conjunction with digital agency Firstborn and director Mark Kudsi. Dubbed “Art of Patrón: Be The Bee,” it puts the viewer in the shoes (or little fuzzy legs) of the brand’s iconic insect as it flies over a tequila field and through the company’s headquarters. It was filmed using an array of GoPro cameras underneath a drone—appropriately enough.
“We put our 360Heros H3Pro7 camera rig on a two foot arm below the drone itself,” explains Denton. “We did that to help with overall visibility, and to minimize how much we would have to replace the sky, or replace ceilings. We also put a camera on top of the drone so we had reference images of what was above, and we could use that to stitch things together.”
The company used Nuke and some custom code to track the footage so CG could be added. But a bigger problem was the drone’s wobbly flight path. “We realized we had to deal with all this movement which was in the drone, which would make people sick,” says Denton. “That’s the main thing you don’t want to do when you’re trying to advertise a product – especially an alcoholic product!”
Denton used the tracking data it had gathered to remove all rotation, resulting in nice smooth movements. But the results were maybe a little too smooth, so the original curves were re-filtered and animations added so that the footage felt natural – but didn’t induce motion sickness.
While The Recruit: R U In and the Art of Patrón film showed that Legend3D could handle live-action and live-action/CG projects respectively, it felt that in order to become a proficient virtual reality studio it needed to tackle a full-CG project. Fortunately, film studio Legendary Pictures was seeking a virtual reality project to market Guillermo del Toro’s upcoming gothic horror Crimson Peak.
“They gave us some of the CG elements effects company Mr. X had done, and said, ‘we have a concept for ComicCon we’d like you to develop.’” says Denton. “But right off the bat I realized that they were set up for V-Ray. I’m traditionally a V-Ray artist as well, so I was excited about that, but the problem was that V-Ray didn’t support stereoscopic 3D rendering then. For me that was a giant bummer.”
V-Ray’s lack of stereoscopic support could have made life a lot more difficult for Denton and Legend3D, but on the off-chance he decided to reach out to Chris Nichols, Chaos Group’s creative director. Virtual reality was, at this point, in the pipeline for 3ds MAX, but thanks to founder Vlado Koylazov Legend3D was allowed access to the nightly test builds of V-Ray for Maya.
“It was very rough early on—there were some big errors and issues,” says Denton. “But I was sending nightly emails, and since they’re on Bulgaria time I would wake up to new responses. It’s amazing how quickly they respond - it blows my mind! You could tell that Vlado and the team were excited about the possibility of it. It went from me not knowing if we were going to be able to use V-Ray, to buying a render farm’s worth of licenses seven days later.”
Particle effects work really well in 3D, and especially in virtual reality, and they were crucial to the Crimson Peak experience. But Phoenix – Chaos Group’s fluid simulations plug-in for Maya – was an unknown quantity when it came to virtual reality. Koylazov was unsure, but granted Denton a test license to try it out.
“It worked beautifully,” says Denton.
My artists and I had zero experience with Phoenix, and it’s the closest to a plug-and-play setup I’ve ever seen for particle sim rendering. I was very impressed with it, the look we were able to get was spot on so fast that we’re definitely Phoenix converts based on this experience!
Justin Denton, Legend3D
The Crimson Peak experience is just the tip of the virtual reality iceberg, and it’s a medium which is experiencing fast growth. Horror is the perfect genre for these simulated worlds because it’s pretty easy to scare people with dark spaces and giant monsters, but Denton believes we’re on the cusp of virtual reality exploding into all forms of entertainment.
“Maybe six months ago we were in the Lumière brothers stage of this, where a train goes by and everybody will buy a ticket to see that in a theatre,” he says. “But we’re rapidly evolving—maybe we’re in the seventies now, I’m not sure! It’s fascinating to see how quickly people are breaking the rules that were set very early on, I think due to the technical limitations, and we’re getting away from that very fast.”
Virtual reality filmmakers, then, are having to create a new ways of using visual language to guide viewers through this brave new world. It’s a medium which is equally inspired by the world of video games as it is films, as there has to be some free choice and interactivity on the viewer’s part.
“A big challenge for directors, especially directors new to the VR space, is ‘how do I make the audience look where I want them to look?’” explains Denton. “We’re coming up with all these methods for keeping the viewer’s eye where you want to go, whether it’s a lighting change that shifts to a certain area, or you have a guide, which can be a person or an animal or a spirit, where you’re physically led to look at a different part of the screen. Those things are going to evolve into the next stage, which will be if you watch a bird fly by, and you stay focused on it long enough, maybe you start to fly with it.”
Of course, the most important part of this virtual reality experience is that people will have access to the headsets. Both the Oculus Rift and Sony’s Project Morpheus are due for release early next year, but there’s a chance that they’ll remain niche, cultish products and sideshow attractions. But Denton hopes not.
“If the Oculus and the Morpheus take off as well as it seems like they might, I feel like it’s going to explode past of just how many devices will be in people’s homes,” he says. “Virtual tourism is going to be really big. The education side of it could be really big. The other big area is scientific R&D. One of the areas where it’s getting really interesting is documentary.” “As a viewer, with a camera that can see everywhere, a doc can’t lie to you and really immerses you in the emotion of a very real moment.”
Next for Denton and Legend3D are more movie tie-ins – but he’s not allowed to say what. But if you’re desperate to try out these uncanny experiences for yourself, you can right here. “We’ll be at the VR Bar at SIGGRAPH,” he says. “We’ll have a number of our latest projects there, which I’m super excited about.”
