Beyond polygons: How Gaussian Splatting transforms 3D rendering

Say hello to Gaussian Splatting (GS) – a game-changing 3D rendering technique that's been turning heads for its jaw-dropping realism and lightning-fast performance! Unlike the old-school methods that rely on polygons or complicated neural networks, GS takes a fresh approach by using millions of tiny, translucent ellipsoids known as ‘Gaussian splats’ to create a scene. Think of each splat as a little pixel with its own personality—packed with info like position, color, size, and transparency. When all these splats come together, they blend so smoothly you’d swear they’re one seamless masterpiece, perfectly capturing the essence of the scene.

How does Gaussian Splatting work?

Creating a Gaussian Splat (GS) model is a bit like a high-tech version of photogrammetry. It all starts by capturing multiple photos or videos of an object or scene from different angles. These images are then analyzed to figure out the camera positions and generate a preliminary 3D point cloud of the scene, a process known as Structure from Motion (SfM). But here's where it gets interesting – instead of sticking with basic points, each one in the cloud gets upgraded into a Gaussian splat. Think of these splats not as simple points, but as tiny ellipsoids, each with its own unique properties, ready to form part of the stunning 3D scene:

• Position: Determines where the splat is located in 3D space.
• Shape and Size: Dictates how the splat stretches or scales, allowing it to represent fine details.
• Color: Stores the splat's color information, which may vary depending on the viewing angle.

Transparency: Defines how transparent the splat is, essential for blending multiple splats.

These splats undergo an optimization process to refine their parameters:

• Differentiable Rasterization: Projects the 3D Gaussian splats onto a 2D image plane, simulating how a camera would see them.
• Loss Function: Measures the difference between the rendered image of splats and the original images.
• Optimization Algorithms: Adjusts the splats to minimize this difference.
• Adaptive Density Control: Removes unnecessary splats or adds more where needed for detail.

The end result? A 3D masterpiece that’s almost identical to the original subject, and it's ready to be rendered in all its glory. The magic happens during the training phase, where the values of the splats are fine-tuned to match the real-life images. The best part? This process is lightning-fast because, unlike neural networks, there are no complex layers to navigate. During training, the splats get to work—some might even split into two to capture extra detail, while others may be removed if their transparency is too low. It's all about perfecting the details in the quickest, most efficient way.

How do I create my own Gaussian Splats?

To create your own GS files, you can follow these steps:

1. Image Capture: Take photos or record videos of the object or scene you want to model from various angles, ensuring good overlap between frames. It is recommended to use wide lenses, high depth of field, and fast shutter speeds to minimize blur.
   
   
2. Processing Software: Use software such as:
   • Polycam: Mobile and web platform for photogrammetry, LiDAR scanning, and GS. It offers editing tools and allows you to upload and process your own images.
   • Luma AI: Platform for processing NeRF and GS, with tools to export camera paths.
   • PostShot: Software for Windows that allows you to train data locally without the need for a command line, offering more control over processing parameters.
   • Nerf Studio: Command-line tool for creating, training, and visualizing 3D Gaussian splats.
   • Scaniverse: Free app that creates splats natively on smartphones, including LiDAR data when available.
3. Training: After uploading your images, the software will use algorithms to convert the point cloud into Gaussian splats and train the model, adjusting the parameters of the splats so that the rendering matches the original images. The process involves differentiable rasterization and optimization.
   
   
4. Export: After training, you can export the GS model. GS files can be exported as a .ply file.

Alright, so maybe you’re not quite ready to dive headfirst into that rabbit hole just yet. No worries! If you're curious about experimenting with GS files, here’s how you can get started without getting too deep into the technical weeds:

• Explore Online Galleries: Many platforms, such as Polycam, offer galleries with examples of Gaussian splats that you can download and view.
• Use Applications and Software: Some of the software mentioned above, such as Polycam and Luma, allow you to view and test splats directly on their platforms.
• Online Communities: Participate in communities and discussion forums to obtain examples of GS files shared by other users.

Uses of Gaussian Splats in architectural visualization

GS is a game-changer for architectural visualization. Thanks to its ability to capture intricate details and stunning lighting effects—like reflections and refractions—it opens up a whole new world of possibilities for creating hyper-realistic renderings of buildings and spaces. Thanks to:

• Realism: GS can generate photorealistic scenes, capturing even the finest details.
• Efficiency: The Gaussian splat representation is more compact than dense polygon meshes or neural networks, resulting in less storage and computational power needs.
• Speed: Real-time or near real-time rendering is possible, making GS suitable for interactive applications.
• Integration: GS can be integrated with computer-generated 3D objects, allowing for the creation of complex and detailed scenes.

Gaussian Splatting in  V-Ray 7

V-Ray 7 is the first commercial ray tracer to support the loading and rendering of Gaussian Splats. This integration opens up new possibilities for artists and designers:

• Ray Tracing: By integrating GS into the V-Ray ray tracing engine, the limitations of realism and flexibility of the original rasterization method are overcome.
• Parallax Effects: Gaussian splats offer precise depth information, resulting in realistic parallax effects when moving the camera.
• View-Dependent Effects: Splats can capture reflections and refractions that change with the viewing angle.
• Occlusion: Gaussian splats can occlude other objects in the scene, allowing for a realistic interaction between elements.
• Depth of Field and Motion Blur: V-Ray can render these effects naturally with Gaussian splats, increasing the photorealism of the scene.

Use as Holdout Objects: Splats can function as holdout objects, interacting with other scene elements without contributing to the final image's RGB and alpha channels.

With these powerful features, V-Ray 7 takes realism to a whole new level, enabling artists and designers to craft scenes with stunningly lifelike backgrounds and environments. By combining computer-generated 3D objects with real-world data represented by Gaussian splats, it bridges the gap between digital and reality.

In a nutshell, Gaussian Splatting marks a major leap forward in 3D rendering, offering a faster and more realistic way to capture and render scenes. When paired with cutting-edge technologies like ray tracing, GS unlocks its full potential, revolutionizing architectural visualization and beyond.