The stage is your world: virtual production technology explained

The evolution of virtual production

As we’ll find, the virtual production process has its roots in some of the earliest techniques in filmmaking. The term virtual cinematography was first coined by John Gaeta, the VFX supervisor on The Matrix, to describe that film’s use of computer-generated imagery (CGI) to create a digital world with photorealistic sets and characters.

From here, computing power increased exponentially, and techniques developed that enabled better-quality visual effects. Real-time software and game engines, such as Chaos Vantage and Epic Games’ Unreal Engine, took advantage of leaps in hardware to bring photorealism to interactive rendering. At the same time, movies such as Gravity and Oblivion used computer-controlled light banks and projected backdrops to blur the lines between visual effects and practical sets.

Inevitably, real-time engines found their way into the Hollywood film production process. Completely CG animated movies and movies with many effects shots, such as The Lion King and Avatar: The Way of Water used virtual reality and virtual cameras to set up and interact with 3D environments and characters as if they were on a real set using real cameras, as well as using performance capture to help actors become animals or aliens in real-time.

The final piece of the puzzle is in-camera visual effects (ICVFX). As the name suggests, these are CG visual effects shots that are captured in-camera. We’ll get into this more later, but by coupling powerful software and hardware with state-of-the-art LED walls, it’s possible to generate realistic, interactive backdrops with which directors, cinematographers, and actors can interact. Examples of TV series and films that have used a virtual production pipeline include The Mandalorian, The Batman, Dune: Part 2, and The House of the Dragon.

Why is virtual production important?

Virtual production solutions, particularly ICVFX, can dramatically alter VFX workflows. A traditional production process involves shooting against a colored screen, typically green or blue, which contrasts against actors’ skin tones. The screen is removed from the shot via a process called chroma keying, and replaces them with CGI in post-production.

Green- or blue-screen production processes produce seamless results, and they’ve become the dominant way to create VFX shots, giving filmmakers and VFX teams incredible flexibility and control over the minutest details. However, they can also result in long production times and drawn-out conversations between VFX supervisors, directors, and cinematographers as they render and re-render shots.

ICVFX turns this process on its head. By shooting on a stage with an LED wall (which can be flat or slightly curved) or an LED volume (which is more immersive), VFX supervisors and artists can create virtual backdrops to film against. What’s more is that they can make changes in the digital world that would be impossible on a practical shoot, such as the position of the sun or the location of a mountain.

In virtual production, this previsualization step is roughly the same, but the construction of the “set” takes place in a virtual environment instead of a physical space. This is handled by a specific team, called the virtual art department, which collaborates with everyone from VFX supervisors to carpenters to ensure a seamless blend of digital and real.

When the set is created in digital space, virtual scouting can take place. Via this process, filmmakers can explore the space with virtual reality headsets and work out the best angles and shots, just like they would with real-world locations. Movies such as The Jungle Book and Avengers: Endgame used real-time engines for on-set previews of how green screen shots would look in the finished movies.

Main virtual production techniques

Use of LED walls

Without good-quality LED screens, ICVFX would fall apart. These fundamentally rely on the same technology that displays imagery on your TV, monitor, or smartphone, with a few key differences:

• No bezels: LED stages are created by building literal walls or volumes of multiple panels that seamlessly link together. Bezels are obviously a no-go in this environment, so LED screens feature edge-to-edge displays. 
• Sturdy and versatile: The panels also have to be tough enough to withstand temperamental directors’ boots and versatile enough to mount in various scenarios, from large flat walls to cavernous volumes, where they can form part of the wall or ceiling.
• Increased brightness: Film sets are surprisingly bright places. A specialized LED screen is much brighter so it's visible on camera, even with studio lights in front of it, and so it can cast realistic lighting and reflections on physical actors and props. It also has a matte finish to avoid unwanted reflections on the screen itself.
• Excellent viewing angles: Have you ever noticed how cheaper monitors can look washed out when viewed from a certain angle? This effect would ruin a virtual production, so most LED panels are designed to be viewed from any angle.
• Low pixel pitch: Likewise, the distance between pixels on the screen (known as pixel pitch) must be as low as possible. If the pixel pitch is too high, there’s a risk of a moire effect when the camera is close to the screen, which can cause unwanted wavy or shimmering lines. Today, 1.8mm is considered a good pitch — but panels used in the production of Tomorrowland in 2015 featured a pitch as high as 11mm!
• Color accuracy and calibration: Spot-on color reproduction is essential for seamlessly blending the LED wall or volume with real-world props and actors. For instance, a virtual desert shot might use real sand on set, so it’s crucial that the colors and lighting match accurately.

Rear projection and virtual sets

Rear projection is nothing new. Those unconvincing shots of actors driving cars against shaky backgrounds in old films made use of this technology by placing a car in front of a projector and replaying footage of a road. Fortunately, it’s come a long way since then — but the fundamental principle is the same.

In an ICVFX production, a car can be placed in front of a moving CG landscape to create a convincing illusion that the vehicle is in motion. However, unlike old rear projection systems, the angle, lighting, and content of the backdrop can be changed if necessary. We used this technique to create the moving steam train footage for “Ray Tracing FTW.”

Camera tracking and motion capture

Virtual production enables a new dimension of rear projection — literally. Because virtual environments are created three-dimensionally, they can be synchronized with the camera, enabling parallax movement. If the camera operator moves the camera up, down, or side-to-side, the background moves as it would in reality, with assets closer to the camera moving faster and distant elements moving more slowly. 

Like the best magic tricks, this illusion is possible thanks to some really clever technology behind the scenes — and deft timing. To synchronize the on-screen display with the camera’s position, the camera itself has to be tracked. There are myriad options for tracking the camera:

• Vicon and OptiTrack are motion capture systems that use reflectors or infrared LEDs, in conjunction with infrared cameras, to track the camera. This is known as outside-in tracking.
• NCAM, Mo-Sys and stYpe work the other way around, using an infrared sensor attached to the camera itself in conjunction with reflectors attached to the ceiling and walls of the set. These are referred to as inside-out tracking systems.
• HTC VIVE Mars Camtrack and SteamVR Tracking repurpose virtual reality headset sensors for use in virtual production.
• Lens metadata, such as focal length, focus, and zoom, can be captured via specialized systems such as Cooke /i Technology or ARRI LDS, and it’s built into stYpe, EZTrack, Mo-Sys, and more.

Each suits different budgets and setups, but they all essentially do the same thing: pass the rendering engine the position of the camera within a CG space so that the screen itself matches the viewpoint of the camera.

The rendering engine output is sent to a media server, such as Brompton Technology or Disguise, which translates it onto the displays. Because the displays have to operate at high frame rates (60hz or above) to ensure smooth playback, it’s critical that everything is synchronized to the split-second, which is where a genlock system needs to be used.

Real-time camera visual effects

Once they’ve lined up their perfect angle, directors, cinematographers, lighting designers, and CG supervisors and artists can tweak just about anything in the virtual world. Want to switch from spooky dusk to harsh midday sun? Or add a layer of snow to the environment? Or literally rearrange a city? With virtual production, it’s easy, and it reduces the need for extensive post-production.

Virtual production tools

Unreal Engine and its role in virtual production

Unreal Engine has become a go-to tool for virtual production. Initially developed for videogames (it’s the same engine that runs Fortnite), Unreal Engine’s photorealistic output, limitless customization and interactivity, realistic physics and particle effects, plus integration with popular tools, have made it a solid choice.

Virtual production is only possible thanks to powerful hardware and software. Unreal Engine’s ability to output the vast, high-resolution imagery required for ICVFX is down to NVIDIA's nDisplay, which uses multiple synchronized graphics processing units (GPUs) to distribute the rendering job.

Other game engines used in virtual production

Unreal Engine rules the roost in virtual production, but Unity, a competing game engine, is also used in more niche applications such as online streaming. Unity doesn’t pack the same feature set as Unreal Engine, but it’s generally considered easier to use. In addition, Industrial Light & Magic, the innovative VFX studio behind Star Wars and Jurassic Park, has its own StageCraft system, which runs on its bespoke Helios rendering engine.

Another option is Chaos Arena. Built on the photorealistic rendering technology Chaos has spent 20 years developing for the VFX industry, Chaos Arena’s advantages are that it works seamlessly with the digital assets VFX artists have already created, and its real-time ray tracing produces physically accurate results.

Just like any other production involving VFX, assets should be categorized and documented accurately so that it’s easy for multiple teams to find them, but fast virtual production workflows often require quick changes and more iterations.

Post-production techniques in virtual production

One of the biggest benefits of virtual production, particularly ICVFX, is the reduction in requirements for post-production. Instead of shooting against a green or blue screen and then waiting weeks or even months for VFX artists and compositors to create and polish shots, they’re almost ready when they come out of the camera. These shots are good enough for editors to work with and require less post-production work.

However, that doesn’t mean post-production is not necessary at all. Computationally expensive effects such as fog, rain, or explosions are often enhanced or added in post, and sometimes the live-action footage capture in production is married to a new CG backdrop. This process very similar to conventional VFX workflows, making use of rendering engines such as V-Ray to deliver high-def imagery.

When motion capture is used, animators are also required to clean up or alter data. And while these productions are “filmed” using real-time game engines and virtual cameras, the final shots are rendered by taking the captured data into 3D software such as Autodesk Maya and then rendering it in an offline rendering engine like V-Ray.

The benefits of virtual production techniques

Cost efficiency

While constructing or hiring an ICVFX stage can be costly, it can save money overall. Instead of flying cast and crew to far-flung locations, they can be built within a virtual production environment. Props, set decorations, background vehicles, aircraft, and even spaceships can also be added with a click of a mouse, avoiding the costs associated with procuring, shipping, and customizing these items.

Time efficiency

Imagine being able to shoot on a city street without waiting for permits. Or capturing the exact icy hue of a South Pole research station exterior while you’re actually in an air-conditioned volume in midsummer LA. Or switching a set from a dusty interrogation room to the shoulder of Orion instantaneously, without waiting for the crew to dismantle and rebuild the set.

With virtual production, artists can also design multiple versions of the same asset, so it can switched out instantaneously when filming begins. 

With virtual production, the time savings can be as big as the cost savings.

Environmental impact

It’s no secret that the traditional film production and industry has an enormous carbon footprint — but, thanks to associations such as the Sustainable Entertainment Alliance, productions are taking steps to reduce their environmental impact. Virtual production can be part of this process; according to virtual production company Quite Brilliant, shooting on a virtual stage produces just 0.74 tonnes of carbon versus a whopping 95 tonnes for a practical shoot.

Another benefit is that real-world locations can be accurately rebuilt in 3D, a process made even easier using 3D Gaussian Splatting. In turn, this reduces environmental damage caused by film crews and allows productions to be shot in ecologically delicate protected areas — albeit virtually.

Improved health and safety

Related are the safety advantages of virtual sets. Want to shoot next to a volcano? Or in outer space? Virtual production makes it easy to place your A-listers in precarious positions without physically endangering them. Virtual production can also create dangerous effects, such as explosions or car crashes, without requiring stunt and special effects crews.