3D Gaussian Splatting: A new frontier in rendering

In the ever-evolving world of 3D graphics, new techniques continually emerge that push the boundaries of what's possible. One such breakthrough is 3D Gaussian Splatting (3DGS), a method that's turning heads for its ability to render highly realistic scenes with remarkable efficiency and speed. 

Using V-Ray 7’s powerful ray tracing capabilities you can now seamlessly blend Gaussian splats of real-life captured environments together with computer generated objects. 

If you're familiar with 3D rendering but not deeply entrenched in the technicalities, read on to discover how 3DGS might be the next big thing in graphics.

What is 3D Gaussian Splatting?

At its core, 3D Gaussian Splatting is a technique for creating and rendering 3D scenes using millions of tiny, translucent ellipsoids known as "Gaussian splats." Unlike traditional methods that rely on polygons or complex neural networks, 3DGS uses these splats to represent a scene. Each splat carries information about its position, color, size, and transparency. When viewed together, they blend seamlessly to replicate the original subject with striking fidelity.

How does 3DGS work?

3D Gaussian Splatting creates an accurate representation of scenes captured from multiple photos taken at various angles, a method widely used in computer graphics. Through a training process involving optimization algorithms and differentiable rasterization, these images are transformed into detailed 3D models ready for rendering.

The process step by step

The process begins much like photogrammetry—you capture a subject from multiple angles using photos or video frames. These images are then analyzed to determine camera positions and create a preliminary 3D point cloud of the scene, a method known as Structure from Motion (SfM).

Each point in this cloud is converted into a Gaussian splat, which isn't just a point but an ellipsoid with specific properties:

• Position: Determines where the splat is located in 3D space.
• Shape and Size: Dictates how the splat stretches or scales, allowing it to represent fine details.
• Color: Stores the splat's color information. The colors are view-dependent and may change depending on the direction that the splat is viewed from.
• Transparency: Defines how transparent the splat is, crucial for blending multiple splats together.

These splats then undergo an optimization process to refine their parameters. This involves:

• Differentiable Rasterization: Projecting the 3D Gaussian splats onto a 2D image plane to simulate how a camera would see them.
• Loss Function: Measuring the difference between the rendered image of splats and the original input images.
• Optimization Algorithms: Adjusting the splats to minimize this difference.
• Adaptive Density Control: Removing unnecessary splats or adding more where needed for detail.

The result is a 3D representation that closely matches the original subject, ready for rendering.

Why is 3D Gaussian Splatting a game-changer?

What makes 3DGS revolutionary is its unique combination of realism, efficiency and speed, positioning it as an exciting advancement in computer graphics.

Realism

3DGS excels at capturing fine details and complex lighting effects like reflections and refractions. This leads to highly photorealistic results that were previously hard to achieve in real-time rendering.

Efficiency

The Gaussian splat representation is more compact than dense polygon meshes or data-heavy neural networks. This means less storage and computational power are needed.

Speed

Because of its efficient data representation and optimized rendering pipeline, 3DGS can achieve real-time or near real-time rendering speeds, making it suitable for interactive applications.

Scalability

It's capable of handling complex scenes with millions of splats, efficiently representing large-scale environments without a significant performance hit.

Rendering 3D Gaussian splats

3DGS can achieve very fast rendering speeds, but this comes at the expense of limitations in realism and flexibility. Integrating Gaussian splats into V-Ray's ray tracing engine overcomes these challenges, allowing for accurate representations of the captured scenes and creative control.

3D Gaussian Splatting, rasterization, and ray tracing

While the Gaussian Splats are relying on the process of rasterization to make them render really fast, this comes at the expense of many limitations. 

Fortunately, one of the exciting developments in 3D Gaussian Splatting is its integration inside of V-Ray, which breaks through these barriers with its powerful ray tracing capabilities.

Rendering Gaussian Splats with V-Ray

V-Ray 7, the first commercial ray tracer to support loading and rendering of Gaussian splats, opens up new possibilities for artists and designers.

In many cases, you might want to use Gaussian splats as a sophisticated background rather than individual objects within a scene. Using V-Ray you can easily place a 3D model in the context of a real location that is converted to a Gaussian Splat. 

Advantages over traditional environment maps

When using Gaussian splats in V-Ray, you gain several benefits over traditional environment maps:

• Proper Parallax Effects: Unlike flat environment maps, Gaussian splats provide accurate depth information, resulting in realistic parallax when the camera moves.
• View-Dependent Effects: The splats can capture reflections and refractions that change based on the viewing angle, enhancing realism.
• Occlusion Handling: Gaussian splats can occlude other objects in the scene, allowing for correct layering and interaction between elements.
• Depth of Field and Motion Blur: V-Ray can render these effects naturally with Gaussian splats, adding to the photorealism of the scene.

Using Gaussian Splats as holdout (matte) objects

In some workflows, you might want the Gaussian splats to function as holdout objects. This means they interact with other scene elements—occluding objects, casting shadows, appearing in reflections and refractions—but they don't contribute to the final RGB and alpha channels of the image.

This setup is useful when you plan to replace the background in post-production with a high-resolution image or video but still want the Gaussian splats to affect lighting and reflections.

Gaussian Splats as individual objects

Gaussian splats can also be used to represent smaller individual objects in the scene. 

Note that the lighting and reflections are baked into the Gaussian splats object and therefore these objects will not be affected by the scene lighting.

Are there limitations?

While promising, 3DGS isn't without its challenges:

• Fine Detail Representation: Extremely fine details may sometimes be lost or appear jagged in the splat representation.
• Memory Requirements: Rendering very large scenes can still require significant memory resources.
• Artistic Control: Editing or manipulating scenes represented by Gaussian splats isn't as straightforward as traditional 3D modeling techniques.

V-Ray 7 introduces rendering of Gaussian splats as scene objects and although the basic functionality is already there, there is still room for improvement.

• Speed Improvements: Although rendering of Gaussian splats is already performant, we believe there are additional improvements that can be made.
• Shadows: The Gaussian Splats are treated as self-illuminated objects in V-Ray and thus can’t receive shadows from other objects directly. Instead you can use a shadow-catcher dummy geometry that covers the place where your shadows should be. 
• Render Elements support: Currently there is no Render element support, but Z-Depth, Cryptomatte, and XYZ passes are in our to do list.
• GPU Rendering: Currently only V-Ray CPU is supported.

Where can you create 3DGS?

Several tools and platforms are beginning to support creation of 3DGS:

• Nerf Studio: A command-line interface and tools for creating, training, and visualizing 3D Gaussian splats.
• Polycam: A web-based renderer using Three.js, allowing for interactive visualization.
• Luma AI: Offers web-based platforms and APIs for generating and rendering high-quality 3D Gaussian splats.
• Postshot: A standalone application for creating, training, and visualizing 3D Gaussian splats.

The future of 3D Gaussian Splatting

The potential applications for 3D Gaussian Splatting are vast:

• Film and Visual Effects: Creating detailed and realistic environments and assets for films, animations, and virtual productions.
• Virtual and Augmented Reality: Enhancing realism and performance in VR/AR experiences.
• Digital Twins and Simulations: Generating accurate and detailed digital representations of real-world objects and environments.

As research continues, we can expect improvements in editing capabilities, better integration with existing workflows, and even more realistic results. 3DGS is poised to become a fundamental technology in the world of 3D graphics and computer vision, unlocking new possibilities for creative expression and technological innovation.

Conclusion

3D Gaussian Splatting is an exciting development in the field of 3D rendering. By offering a more efficient and realistic way to capture and render scenes, it holds the promise of transforming various industries—from film to architectural visualizations to virtual reality. While it's still a developing technology with some limitations, its advantages make it a technique worth exploring.

Whether you're a 3D artist looking for new tools or just someone fascinated by the latest in rendering technology, 3DGS represents a significant step forward in how we create and experience digital worlds.