3D Gaussian Scene Encoding for Standardized XR Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing scene description formats like MPEG-I SD do not support the standardized representation and encoding of 3D Gaussian splatting models, limiting the ability to efficiently render and interact with such models in extended reality applications.
Innovation Solution
The proposed solution involves extending the glTF format to include new attributes such as COVARIANCE, SH0_n, SH1_n, SH2_n, and ALPHA to encode 3D Gaussian models, using spherical harmonics for color representation and adding these attributes directly or through an 'MPEG_primitive_gaussian' extension to support 3D Gaussian splatting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing scene description formats like MPEG-I SD are used, then compatibility with current standards is maintained, but support for 3D Gaussian splatting models is lacking
Solution Approach 1:
The patent embeds 3D Gaussian model data within existing glTF scene description structures by nesting Gaussian-specific attributes (position, covariance, spherical harmonics coefficients, alpha) inside primitive elements. This allows the new format to contain the old format while adding specialized support for Gaussian splatting without breaking existing compatibility.
Solution Approach 2:
The extended glTF format is designed to serve multiple functions: it can represent traditional mesh-based 3D content while simultaneously supporting 3D Gaussian splatting models. The covariance attribute and spherical harmonics coefficients enable the same data structure to handle both conventional geometry and Gaussian-based radiance fields.
2Productivity
If new attributes are added to encode 3D Gaussian models, then rendering capability is improved, but data structure complexity increases
Solution Approach 1:
The patent segments the 3D Gaussian model representation into distinct attribute categories: position attributes (first attribute), covariance attributes (second attribute, stored as 3x3 matrix), color attributes (third, fourth, and fifth attributes corresponding to spherical harmonics coefficients), and transparency attributes (sixth attribute, alpha coefficient). This segmentation allows each attribute type to be processed independently during rendering, improving efficiency while maintaining organized data structures.
Solution Approach 2:
The patent transforms the representation of 3D Gaussian models by introducing specific parameters: covariance matrices to define shape and orientation, spherical harmonics coefficients (multiple sets) to encode color information under different lighting conditions, and alpha coefficients for transparency. These parameter changes enable efficient rendering by pre-computing and storing all necessary visual properties.
3Ease of manufacture
If covariance attributes are stored within glTF primitive elements, then integration is simplified, but flexibility for future extensions is reduced
Solution Approach 1:
The patent nests covariance attributes and other Gaussian-specific data within the existing glTF primitive data structure elements. This nesting approach simplifies integration by leveraging the established glTF parsing and processing pipeline, while the nested structure allows future Gaussian-related extensions to be added without disrupting the core glTF format.
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
Some embodiments of a method may include: obtaining information for a three-dimensional (3D) Gaussian model corresponding to a 3D scene, wherein the information comprises a set of attributes of the 3D Gaussian model; parsing the information for a first attribute of the set of attributes, wherein the first attribute corresponds to a position of the 3D Gaussian model; parsing the information for a second attribute of the set of attributes, wherein the second attribute corresponds to a covariance of the 3D Gaussian model; parsing the information for third, fourth, and fifth attributes, wherein the third, fourth, and fifth attributes correspond to first, second, and third sets of spherical harmonics coefficients associated with the 3D Gaussian model; parsing the information for a sixth attribute of the set of attributes, wherein the sixth attribute is an alpha coefficient for the 3D Gaussian model; and rendering the 3D scene using the parsed attributes.