3D digital human and scene processing method and apparatus, and device, medium and product

Abstract

Provided in the present disclosure are a 3D digital human and scene processing method and apparatus, and a device, a medium and a product. The method comprises: acquiring 3D data based on 3D Gaussian splatting (3DGS); performing representation generation on the 3D data, specifically comprising: processing the 3D data to obtain a reference point position and a reference point attribute of each reference point, and on the basis of the reference point positions and the reference point attributes combined with Gaussian sphere positions correspondingly generated for the reference points, performing feature transformation to obtain an attribute context feature of each reference point; using the attribute context features to quantize the reference point attributes of the reference points; and on the basis of the attribute context features, performing entropy encoding on the quantized reference point attributes to obtain an encoding result.

Description

Methods, devices, equipment, media, and products for processing 3D digital humans and scenes.

[0001] Cross-references to related applications

[0002] This disclosure claims priority to Chinese Patent Application No. 202411988755.7, filed in China on December 31, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the technical field of video transmission, and more specifically, to a method, apparatus, device, medium, and product for processing 3D digital humans and scenes. Background Technology

[0004] Digital humans are virtual characters created using computer technology and artificial intelligence, exhibiting human appearance or behavior. These virtual characters have a wide range of applications, including corporate promotion, education and training, virtual actors and hosts, intelligent customer service, virtual exhibitions and tour guides, IP character customization, and virtual conferences. Recently, 3D digital humans have become a research hotspot, boasting advantages such as high realism, flexibility, and cost-effectiveness, and are expected to find widespread application in entertainment, education, commerce, and science.

[0005] In related technologies, 3D Gaussian Splatting (3DGS) can be used to represent 3D digital humans. However, this representation method requires a large number of 3D Gaussian functions to accurately represent large-scale scenes. Consequently, each point and its attributes are stored independently, inevitably leading to significant storage costs. For example, a typical 3DGS representation of a 360-degree scene requires millions of Gaussian spheres, exceeding 1GB of storage space. Therefore, the problem of compressing 3DGS-based digital humans transforms into the problem of efficiently compressing Gaussian spheres and their associated attributes. Summary of the Invention

[0006] This disclosure provides at least one method, apparatus, device, medium, and product for processing 3D digital humans and scenes.

[0007] In a first aspect, embodiments of this disclosure provide a method for processing 3D digital humans and scenes, including:

[0008] Acquire 3D data based on 3D Gaussian sputtering 3DGS;

[0009] The 3D data is characterized, generated, quantized, and / or entropy encoded.

[0010] Secondly, embodiments of this disclosure provide a method for processing 3D digital humans and scenes, including:

[0011] Acquire 3D data based on 3D Gaussian sputtering 3DGS;

[0012] The 3D data is characterized and generated, specifically including: processing the 3D data to obtain the reference point position and reference point attribute of each reference point, and performing feature transformation based on the reference point position and the reference point attribute, combined with the Gaussian sphere position generated corresponding to the reference point, to obtain the attribute context feature of each reference point.

[0013] The reference point attributes of the reference point are quantified using the attribute context features.

[0014] The quantized baseline attribute is entropy encoded based on the attribute context features to obtain the encoding result.

[0015] In one optional implementation, processing the 3D data to obtain the reference point position and reference point attributes of each reference point includes:

[0016] The 3D data is processed to generate reference points and predict attributes, thereby obtaining the reference point position and reference point attributes of each reference point.

[0017] In one optional implementation, the step of generating reference points and predicting attributes from the 3D data to obtain the reference point position and reference point attributes of each reference point includes:

[0018] The 3D data is converted into voxel mesh data;

[0019] Based on the position of the center point of each voxel in the voxel mesh data, the position of the reference point is determined, and the reference point attribute of the reference point corresponding to the position of the reference point is determined.

[0020] In one optional implementation, the step of performing feature transformation based on the reference point position and the reference point attributes, combined with the Gaussian sphere position generated corresponding to the reference point, to obtain the attribute context features of each reference point includes:

[0021] Based on the location and attributes of the reference point, determine the locations of multiple Gaussian spheres corresponding to the reference point;

[0022] Based on each Gaussian sphere position, the reference point position, and the reference point features, the attribute context features of the reference point are determined.

[0023] In one optional implementation, determining the positions of multiple Gaussian spheres corresponding to the reference point based on the reference point position and the reference point attributes includes:

[0024] Calculate the product between the learnable offset in the reference point attribute and the scaling factor in the reference point attribute;

[0025] The positions of the Gaussian spheres are obtained by summing the product and the reference point position.

[0026] In one optional implementation, determining the attribute context features of the reference point based on each Gaussian sphere position, the reference point position, and the reference point features includes:

[0027] Based on the position of each Gaussian sphere, a predetermined number of target Gaussian spheres that are closest to the reference point are determined from among the plurality of Gaussian spheres;

[0028] Based on the Gaussian sphere position of the target Gaussian sphere, the reference point position, and the reference point characteristics, the attribute context features of the reference point are determined.

[0029] In one optional implementation, determining the attribute context features of the reference point based on the Gaussian sphere position of the target Gaussian sphere, the reference point position, and the reference point features includes:

[0030] Based on the position of the target Gaussian sphere, the distance between the target Gaussian sphere and the reference point is determined, resulting in multiple first distances;

[0031] Summing the reciprocals of each of the first distances yields the summation result;

[0032] The product between the summation result and the feature of the reference point is calculated to obtain the attribute context feature of the reference point.

[0033] In one optional implementation, quantizing the reference point attributes of the reference point using the attribute context features includes:

[0034] The target quantization step size of the reference point attribute is determined using the attribute context features.

[0035] The reference point attributes are quantized using the target quantization step size to obtain the quantized reference point attributes.

[0036] In one optional implementation, determining the target quantization step size of the reference point attribute includes:

[0037] The attribute context features are processed by a first multilayer perceptron to obtain quantization parameters;

[0038] The target quantization step size is obtained by adjusting the predefined quantization step size using the quantization parameters.

[0039] In one optional implementation, the step of entropy encoding the quantized reference point attribute based on the attribute context features to obtain the encoding result includes:

[0040] The probability distribution of the reference point attribute is determined based on the attribute context features and the quantized reference point attribute.

[0041] The baseline point attributes are entropy-encoded using the probability distribution to obtain the encoding result.

[0042] In one optional implementation, determining the probability distribution of the reference point attribute based on the attribute context features and the quantized reference point attribute includes:

[0043] The attribute context features are processed by the second multilayer perceptron to obtain the Gaussian distribution parameters of each sub-attribute in the baseline attribute;

[0044] The probability distribution of the reference point attribute is calculated based on the Gaussian distribution parameters, the quantized reference point attribute, and the target quantization step size of the reference point attribute.

[0045] Thirdly, embodiments of this disclosure also provide a processing apparatus for 3D digital humans and scenes, including:

[0046] The acquisition module is used to acquire 3D data based on 3D Gaussian sputtering 3DGS.

[0047] The processing module is used to perform characterization, quantization, and / or entropy encoding on the 3D data.

[0048] Fourthly, embodiments of this disclosure also provide a processing apparatus for 3D digital humans and scenes, including:

[0049] The acquisition unit is used to acquire 3D data based on 3D Gaussian sputtering 3DGS.

[0050] The characterization generation unit is used to generate characterizations of the 3D data, specifically including: processing the 3D data to obtain the reference point position and reference point attribute of each reference point, and performing feature transformation based on the reference point position and the reference point attribute, combined with the Gaussian sphere position generated corresponding to the reference point, to obtain the attribute context features of each reference point.

[0051] A quantization unit is used to quantize the reference point attributes of the reference point using the attribute context features;

[0052] An entropy coding unit is used to entropy code the quantized reference point attribute according to the attribute context features to obtain the coding result.

[0053] Fifthly, embodiments of this disclosure also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, they perform the steps of the first aspect, or the second aspect, or any possible implementation of the first aspect.

[0054] In a sixth aspect, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect, or the second aspect, or any possible implementation of the first aspect.

[0055] In a seventh aspect, embodiments of this disclosure also provide a computer program product stored in a storage medium, the program product being executed by at least one processor to perform the steps of the first aspect, or the second aspect, or any possible implementation of the first aspect.

[0056] This disclosure provides a method, apparatus, device, medium, and product for processing 3D digital humans and scenes. In embodiments of this disclosure, after acquiring 3D data based on 3D Gaussian sputtering 3DGS, the 3D data is processed to obtain the reference point position and reference point attributes for each reference point. Based on the reference point position and the reference point attributes, and combined with the position of the Gaussian sphere corresponding to the reference point, feature transformation is performed to obtain the attribute context features of each reference point. Next, the reference point attributes of the reference points are quantized using the attribute context features. Finally, entropy encoding is performed on the quantized reference point attributes according to the attribute context features to obtain the encoding result.

[0057] In the above embodiments, by processing 3D data to obtain the reference point attributes and reference point positions, and determining the corresponding Gaussian sphere positions, it is not necessary to transmit all Gaussian sphere information, thus reducing the amount of data transmitted for encoding; by using attribute context features to entropy encode the quantized reference point attributes, the efficiency of the reference point attribute entropy encoding process can be improved.

[0058] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0060] Figure 1 shows a flowchart of a method for processing 3D digital humans and scenes provided in an embodiment of this disclosure;

[0061] Figure 2 shows a system flowchart of a 3D digital human and scene processing method provided in an embodiment of this disclosure;

[0062] Figure 3 shows a schematic diagram of a 3D digital human and scene processing device provided in an embodiment of the present disclosure;

[0063] Figure 4 shows a schematic diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0065] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0066] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0067] As described above, 3DGS representation requires a large number of 3D Gaussian functions to accurately represent large-scale scenes. Therefore, each point and its attributes are stored independently, inevitably leading to significant storage costs. For example, a typical 3DGS representation of a 360-degree scene requires millions of Gaussian spheres, exceeding 1GB of storage space. Therefore, the problem of compressing 3DGS-based digital humans is transformed into the problem of efficiently compressing Gaussian spheres and their related attributes.

[0068] Based on the above research, this disclosure provides a method, apparatus, device, medium, and product for processing 3D digital humans and scenes. In the embodiments of this disclosure, after acquiring 3D data based on three-dimensional Gaussian sputtering 3DGS, the 3D data is characterized and generated, specifically including: processing the 3D data to obtain the reference point position and reference point attributes of each reference point; and performing feature transformation based on the reference point position and the reference point attributes, combined with the corresponding Gaussian sphere position, to obtain the attribute context features of each reference point; next, quantizing the reference point attributes using the attribute context features; and finally, performing entropy encoding on the quantized reference point attributes according to the attribute context features to obtain the encoding result.

[0069] In the above embodiments, by processing 3D data to obtain the reference point attributes and reference point positions, and determining the corresponding Gaussian sphere positions, it is not necessary to transmit all Gaussian sphere information, thus reducing the amount of data transmitted for encoding; by using attribute context features to entropy encode the quantized reference point attributes, the efficiency of the reference point attribute entropy encoding process can be improved.

[0070] To facilitate understanding of this embodiment, a detailed description of a 3D digital human and scene processing method disclosed in this disclosure is provided first. The execution entity of the 3D digital human and scene processing method provided in this disclosure is generally an electronic device with a certain computing power. In some possible implementations, the 3D digital human and scene processing method can be implemented by a processor calling computer-readable instructions stored in memory. This disclosure embodiment can be applied to the data compression process of 3D digital humans and related scenes (e.g., scenes). The related scenes include 3D objects, 3D spaces, and other related three-dimensional scenes.

[0071] This disclosure provides a method for processing 3D digital humans and scenes, including:

[0072] Acquire 3D data based on 3D Gaussian sputtering 3DGS;

[0073] The 3D data is characterized, generated, quantized, and / or entropy encoded.

[0074] In one optional implementation, the process of characterizing, quantizing, and / or entropy encoding the 3D data includes: characterizing, transforming, quantizing, and / or entropy encoding the 3D data.

[0075] In one optional implementation, the process of characterizing, quantizing, and / or entropy encoding the 3D data includes: characterizing, transforming, quantizing, and entropy encoding the 3D data.

[0076] In an optional implementation, the characterization, quantization, and / or entropy encoding of the 3D data includes:

[0077] The 3D data is characterized and generated, specifically including: processing the 3D data to obtain the reference point position and reference point attribute of each reference point, and performing feature transformation based on the reference point position and the reference point attribute, combined with the Gaussian sphere position generated corresponding to the reference point, to obtain the attribute context feature of each reference point.

[0078] The reference point attributes of the reference point are quantified using the attribute context features.

[0079] The quantized baseline attribute is entropy encoded based on the attribute context features to obtain the encoding result.

[0080] Referring to Figure 1, which is a flowchart of a method for processing 3D digital humans and scenes according to an embodiment of this disclosure, the method includes steps S101 to S104, wherein:

[0081] S101: Acquire 3D data based on 3D Gaussian sputtering 3DGS.

[0082] Here, 3DGS uses a set of anisotropic 3D Gaussian functions to represent the scene. These functions have differential properties of volume representation and can efficiently render the scene through tile-based rasterization.

[0083] 3D Gaussian sputtering is an emerging method for representing 3D digital humans. 3DGS models the 3D world as a mixture of multiple 3D Gaussian distributions; theoretically, this mixture can fit any distribution. 3D Gaussian sputtering models the shape of objects in the scene, while 3DGS uses spherical harmonics for object color modeling. Spherical harmonics can express the magnitude of a sphere at various points, representing the RGB value at the corresponding location. In summary, shape is explicitly represented by 3D Gaussian sputtering, and color is represented by spherical harmonics. Through projection relationships, a visualized image of the 3D scene can be rendered from any viewpoint. 3DGS can achieve high-quality real-time rendering at 1080p resolution and brings revolutionary improvements in scene control and editability.

[0084] S102: The 3D data is characterized and generated, specifically including: processing the 3D data to obtain the reference point position and reference point attribute of each reference point, and performing feature transformation based on the reference point position and the reference point attribute, combined with the Gaussian sphere position generated corresponding to the reference point, to obtain the attribute context feature of each reference point.

[0085] In this embodiment of the disclosure, the 3D data can be processed by the characterization generation module to obtain the reference point position and reference point attributes of each reference point. Based on the reference point position and reference point attributes, and combined with the Gaussian sphere position generated corresponding to the reference point, feature transformation is performed to obtain the attribute context features of each reference point.

[0086] Here, the sparse point cloud acquired from 3D digital human signal acquisition can be used as input to the representation generation module. This module generates sparse voxel mesh data based on the sparse point cloud, thereby determining a reference point based on the center point of each voxel, and defining the reference point's location and attributes. Reference point attribute A includes the following: reference point features. Scaling factor l v ∈R 3 and k learnable offsets O v ∈R 3k , where D v It is a vector f v Dimensions.

[0087] For each reference point, at least one Gaussian sphere can be generated. In this embodiment, the distance between the reference point and the corresponding Gaussian sphere can be weighted with the reference point features of that reference point to obtain attribute context features. These attribute context features can be used to predict the probability distribution of the reference point's attributes, thus assisting in the subsequent entropy coding probability distribution and achieving better coding results.

[0088] S103: Quantify the reference point attributes of the reference point using the attribute context features.

[0089] To facilitate entropy coding, the value of the reference point attribute A needs to be quantized into a finite set. This disclosed technical solution does not limit the quantization method; for example, various methods such as rounding quantization algorithms and adaptive quantization algorithms can be used. No specific limitation is made to the quantization algorithm here, only those methods that can be implemented.

[0090] S104: Perform entropy encoding on the quantized reference point attributes based on the attribute context features to obtain the encoding result.

[0091] In this embodiment of the disclosure, the probability distribution of the reference point attribute can be determined based on the attribute context features, and then the reference point attribute can be entropy encoded based on the probability distribution to obtain the encoding result.

[0092] In the above embodiments, by processing 3D data to obtain the reference point attributes and reference point positions, and determining the corresponding Gaussian sphere positions, it is not necessary to transmit all Gaussian sphere information, thus reducing the amount of data transmitted for encoding; by using attribute context features to entropy encode the quantized reference point attributes, the efficiency of the reference point attribute entropy encoding process can be improved.

[0093] The above process will be described in detail below with reference to specific implementation methods.

[0094] As described above, in this embodiment of the disclosure, after acquiring 3D data based on 3D Gaussian sputtering 3DGS, the 3D data can be processed by a characterization generation module to obtain the reference point position and attributes of each reference point. This characterization generation module includes the following modules: a reference point generation and attribute prediction module and a feature transformation module.

[0095] Based on this, the above steps process the 3D data to obtain the reference point position and reference point attributes of each reference point, specifically including the following steps:

[0096] The 3D data is processed to generate reference points and predict attributes, thereby obtaining the reference point position and reference point attributes of each reference point.

[0097] In this embodiment of the disclosure, after obtaining 3D data, the 3D data can be processed to perform reference point generation attribute prediction processing on the processing results, thereby obtaining the reference point position and reference point attribute of each reference point.

[0098] In this embodiment of the disclosure, the above steps perform reference point generation and attribute prediction processing on the 3D data to obtain the reference point position and reference point attributes of each reference point, specifically including:

[0099] Step S11: Convert the 3D data into voxel mesh data;

[0100] Step S12: Based on the position of each voxel center point in the voxel mesh data, determine the position of the reference point and the reference point attribute of the reference point corresponding to the position of the reference point.

[0101] In this embodiment, the structure from motion (SFM) method is used to generate voxel mesh data of 3D data; wherein, the center point of each voxel can be used as a reference point v (anchor), that is, the position of each center point determines the reference point position of the anchor point; wherein, each anchor point has anchor point features. Scaling factor l v ∈R 3 and k learnable offsets O v ∈R 3k These are the attributes.

[0102] After obtaining the reference point location and attributes of each reference point, it is also necessary to determine the locations of the multiple Gaussian spheres corresponding to the reference point and the attribute characteristics of the Gaussian spheres.

[0103] Specifically, for position x c Observation point (camera) and location x v The relative distance and viewing direction between the reference points can be calculated using the following formula:

[0104] The relative distance and viewing direction are used to determine the property characteristics of the Gaussian sphere corresponding to the reference point. The property characteristics of the Gaussian sphere are used to render the three-dimensional scene and the visualization image of the 3D digital human from any viewing angle during the rendering process of the 3D digital human.

[0105] In the above-mentioned process of generating reference points and predicting attributes, the center of each voxel in the above voxel grid data is used as a reference point; each reference point is connected to a set of Gaussian functions with learnable offsets, and its attribute features are further dynamically predicted based on the reference point features and viewpoint position.

[0106] In an optional implementation, the above steps, based on the reference point location and the reference point attributes, and combined with the Gaussian sphere location generated corresponding to the reference point, perform feature transformation to obtain the attribute context features of each reference point, specifically including the following steps:

[0107] Step S21: Based on the location and attributes of the reference point, determine the positions of multiple Gaussian spheres corresponding to the reference point;

[0108] Step S22: Determine the attribute context features of the reference point based on each Gaussian sphere position, the reference point position, and the reference point features.

[0109] In this embodiment of the disclosure, after determining the reference point, a 3DGS corresponding to the scene (i.e., a 3D Gaussian sphere corresponding to the reference point) can be generated from the reference point, and the attribute features of each 3D Gaussian sphere can be determined. The parameterization of the Gaussian function can be achieved through these attribute features. Specifically, the attribute features of the 3D Gaussian sphere include the following: opacity α∈R, covariance-related quaternion q∈R. 4 Scaling s∈R 3 and color c∈R 3 .

[0110] Here, the position of the Gaussian sphere corresponding to the reference point can be determined by combining the reference point's location and some attributes from its properties. Using individual multilayer perceptrons (MLPs), the reference point's features f can be directly used to determine the position. v The relative viewing distance δ between the camera and the reference point vc and perspective direction Decoding yields the attribute features of k 3D Gaussian spheres. The MLP used to calculate each attribute feature can be denoted as F. α F c F q and F s These are used to decode and obtain opacity, color, quaternion, and scaling features, respectively.

[0111] For example, the opacity value of a 3D Gaussian sphere generated from a reference point can be calculated using the following formula:

[0112] Other properties of a 3D Gaussian sphere, such as color {c i}、 Quaternion {q i} and scaling features {s i All of these are derived in a similar way, meaning the formula can be expressed as: and

[0113] After determining the position of the Gaussian sphere in the above manner, the distance between the reference point and the corresponding Gaussian sphere can be weighted with the reference point feature of the reference point to obtain the attribute context feature.

[0114] In this embodiment of the disclosure, the above steps, based on the location and attributes of the reference point, determine the positions of multiple Gaussian spheres corresponding to the reference point, specifically including:

[0115] First, calculate the product between the learnable offset in the reference point attribute and the scaling factor in the reference point attribute;

[0116] Next, the product and the reference point position are summed to obtain the positions of each Gaussian sphere.

[0117] As described above, the technical solution disclosed herein generates k 3D Gaussian spheres for each visible reference point within the view frustum and predicts their attribute characteristics.

[0118] Here, it is necessary to determine the k 3D Gaussian spheres corresponding to the reference point by combining the reference point's location and attributes. Specifically, given a point located at x... v The reference point, and the formula for calculating the position of its 3D Gaussian sphere, are described below:

[0119] {μ0,...,μ k-1}=x v +{O0,...,O k-1}·l v ; where {O0,O1,...,O k-1}∈R k×3 These are learnable offsets (k offsets), l v It is the scaling factor associated with that reference point.

[0120] Here, firstly, the learnable offset O in the reference point attribute is calculated. v With scaling factor l v The product between {O0,...,O k-1}·l v Then, the sum of this product and the reference point position is calculated to obtain the position {μ0,...,μ} of each Gaussian sphere in the k 3D Gaussian spheres. k-1}

[0121] In this embodiment of the disclosure, the above steps, based on the position of each Gaussian sphere, the position of the reference point, and the features of the reference point, determine the attribute context features of the reference point, specifically including:

[0122] First, based on the position of each Gaussian sphere, a predetermined number of target Gaussian spheres that are closest to the reference point are determined from among the plurality of Gaussian spheres;

[0123] Secondly, based on the Gaussian sphere position of the target Gaussian sphere, the reference point position, and the reference point characteristics, the attribute context features of the reference point are determined.

[0124] After the benchmark point generation and attribute prediction are completed, the distance between the Gaussian sphere generated corresponding to the benchmark point and the benchmark point itself can be compared with the benchmark point feature f. v The weighted result is the attribute context feature f. h This information is then used as context to predict the probability distribution of baseline attributes, thereby assisting in subsequent entropy coding of the probability distribution to achieve better coding results.

[0125] As described above, each reference point is determined by its position x. v ∈R 3 and reference point attributes The system is composed of several components, where each component in the reference point attribute represents the reference point feature, scaling factor, and learnable offset, respectively. For each reference point, k Gaussian spheres are generated. Here, m Gaussian spheres closest to the reference point can be selected as target Gaussian spheres, where m = [k / 3]. Then, based on the positions of the target Gaussian spheres, the reference point position, and the reference point features, the attribute context feature f is determined. h Specifically, the distance between the target Gaussian sphere and the reference point can be correlated with the feature f of the reference point. v The weighted result is the attribute context feature f. h .

[0126] In this embodiment of the disclosure, the above steps, based on the Gaussian sphere position of the target Gaussian sphere and the reference point position and features, determine the attribute context features of the reference point, specifically including the following steps:

[0127] First, based on the position of the target Gaussian sphere, the distance between the target Gaussian sphere and the reference point is determined, resulting in multiple first distances;

[0128] Next, sum the reciprocals of each of the first distances to obtain the summation result;

[0129] Finally, the product between the summation result and the reference point feature is calculated to obtain the attribute context feature of the reference point.

[0130] In this embodiment of the disclosure, the distance from the target Gaussian sphere to the reference point (i.e., the first distance) can be determined based on the position of the target Gaussian sphere and the position of the reference point. This first distance is denoted as d1, d2, d3, ..., d... m Next, calculate the reciprocal of each first distance, which are respectively Then, sum the reciprocals of the first distance to obtain the summation result. Finally, the product of this summation result and the baseline feature can be calculated. Thus, the attribute context feature f is obtained. h .

[0131] Therefore, the above process can be described by a formula:

[0132] In this disclosed technical solution, the attribute context features are obtained by weighting the distance between the generated Gaussian sphere and the reference point by combining the reference point location and reference point features, and then using them as context to predict the probability distribution of the reference point attributes, which can improve the efficiency of subsequent entropy coding.

[0133] In this embodiment of the disclosure, after obtaining the attribute context features through the characterization generation module, the reference point attributes of the reference point can be quantified using the attribute context features, specifically including the following steps:

[0134] Step S31: Determine the target quantization step size of the reference point attribute using attribute context features;

[0135] Step S32: Quantize the reference point attributes using the target quantization step size to obtain the quantized reference point attributes.

[0136] Here, after obtaining the attribute context feature f h Next, we need to utilize the attribute context feature f h Minimize the entropy of the reference point attribute A, thereby reducing the number of bits when encoding the reference point attribute using entropy coding.

[0137] To facilitate entropy coding, the value of the reference point attribute A can be quantized into a finite set. This disclosure does not limit the quantization method; for example, traditional rounding, self-adaptive quantization, and other methods can be used. An adaptive quantization algorithm is introduced below.

[0138] In this embodiment of the disclosure, the adaptive quantization algorithm can adaptively determine the quantization step size using an MLP to obtain a target quantization step size. Then, the reference point attributes can be quantized using the target quantization step size to obtain the quantized reference point attributes.

[0139] In this embodiment of the disclosure, the above steps for determining the target quantization step size of the reference point attribute specifically include:

[0140] The attribute context features are processed by a first multilayer perceptron to obtain quantization parameters;

[0141] The target quantization step size is obtained by adjusting the predefined quantization step size using the quantization parameters.

[0142] In the aforementioned adaptive quantization method, MLPq (i.e., the first multilayer perceptron) can be used to estimate the quantization parameters. During the quantization process, the context model MLPq and attribute context features f are utilized. i h The input is used to predict the quantization parameter r. i ∈R 1 The specific formula can be described as r i =MLP q (f i h ); where the quantization parameter is used to adjust the predefined quantization step size Q0.

[0143] After obtaining the quantization parameter r i Then, the predefined quantization step size Q0 is adjusted according to this quantization parameter to obtain the target quantization step size. The specific adjustment formula is described as follows: q i =Q0×(1+Tanh(r) i )).

[0144] After quantizing the reference point attributes, the quantized reference point attributes can be entropy encoded based on the attribute context features to obtain the encoding result. The specific steps include the following:

[0145] Step S41: Determine the probability distribution of the reference point attribute based on the attribute context features and the quantized reference point attribute;

[0146] Step S42: Entropy encoding is performed on the reference point attributes using the probability distribution to obtain the encoding result.

[0147] In this embodiment of the disclosure, after obtaining the target quantization step size q i Then, the baseline point attributes can be quantized according to the target quantization step size to obtain the quantized baseline point attributes.

[0148] Here, an MLP can be used to model the Gaussian parameters, denoted as MLP. c Its structure can be compared with the quantized context model MLP. q Same. Contextual Model MLP using this Gaussian parameter. c It can be used to determine the probability distribution of baseline attributes. Specifically, this Gaussian parameter context model MLP c This is used to model Gaussian parameters, thereby obtaining the probability distribution of the baseline point attributes.

[0149] Next, the reference point attributes can be entropy-encoded using this probability distribution to obtain the encoding result. Using this probability distribution to assist in the entropy encoding of the quantized reference point attributes can achieve better encoding results.

[0150] In this embodiment of the disclosure, the above steps, which determine the probability distribution of the reference point attribute based on the attribute context features and the quantized reference point attribute, specifically include:

[0151] First, the attribute context features are processed by the second multilayer perceptron to obtain the Gaussian distribution parameters of each sub-attribute in the baseline attribute;

[0152] Secondly, the probability distribution of the reference point attribute is calculated based on the Gaussian distribution parameters, the quantized reference point attribute, and the target quantization step size of the reference point attribute.

[0153] Based on the above description, the attributes of a benchmark point include: benchmark point features. Scaling factor l v ∈R 3 and k learnable offsets O v ∈R 3k Here, the reference point features, scaling factor, and offset are the sub-attributes described in the above steps. The second multilayer perceptron is the Gaussian parameter context model MLP described above. c .

[0154] All three sub-attributes of the benchmark attribute A exhibit a Gaussian distribution statistical trend. However, using the same set of Gaussian distribution parameters μ and σ for different sub-attributes may lack accuracy. Therefore, the technical solution disclosed in this paper assumes that the values ​​of each sub-attribute of the benchmark attribute A are independent of each other. In this case, it is necessary to determine the Gaussian distribution parameters for each sub-attribute separately. The Gaussian distribution parameters μ and σ for each sub-attribute are based on the context model MLP. c From f h It was estimated.

[0155] After obtaining the Gaussian distribution parameters for each sub-attribute, the probability density function of the benchmark point can be determined based on these parameters. and cumulative distribution function Next, we can use this probability density function... and cumulative distribution function The quantized baseline attribute and the target quantization step size of the baseline attribute are used to calculate the probability distribution of the baseline attribute.

[0156] Specifically, for the i-th reference point and its quantized reference point attribute vector The Gaussian distribution parameters for each sub-attribute can be calculated using the following formula: μ i ,σ i =MLP c (f i hThen, using the estimated μ i ∈R D and σ i ∈R D It can be calculated The probability of is calculated using the following formula:

[0157] As described above, the technical solution disclosed herein utilizes a Gaussian parameter context model (MLP). c The attribute context features are processed to obtain the Gaussian distribution parameters of each benchmark attribute. Then, using the Gaussian distribution parameters, the quantized attribute context features, and the target quantization step size, the probability distribution of the benchmark attribute is obtained, and entropy encoding is performed on the benchmark attribute based on this probability distribution. This processing method can improve the efficiency of the benchmark attribute entropy encoding process.

[0158] The steps described above will be explained below with reference to Figure 2. Figure 2 shows the system flowchart of the 3D digital human and scene processing method. As shown in Figure 2, this method mainly includes the following processes: representation generation, quantization, and entropy encoding; wherein, representation generation includes SFM processing, reference point generation and attribute prediction, and feature transformation; the specific implementation process of the above processes is described below:

[0159] The process of characterization generation:

[0160] First, the Structure for Motion Recovery (SFM) method is used to generate sparse voxel mesh data of 3D data. The center point of each voxel can be regarded as a reference point v (anchor), which has the characteristics of a reference point. Scaling factor l v ∈R 3 and k learnable offsets O v ∈R 3k .

[0161] Specifically, for position x c Observation point (camera) and location x v The relative distance and viewing direction between the reference points can be calculated using the following formula:

[0162] The relative distance and viewpoint direction are used to determine the property characteristics of the Gaussian sphere corresponding to the reference point. These property characteristics are used in the rendering process of the 3D digital human.

[0163] After determining the reference point, a 3DGS corresponding to the scene (i.e., a 3D Gaussian sphere corresponding to that reference point) is generated from the reference point, and the attribute features of each 3D Gaussian sphere are determined. These attribute features allow for the parameterization of the Gaussian function. Specifically, the attribute features of the 3D Gaussian sphere include the following: opacity α∈R, covariance-related quaternion q∈R. 4 Scaling s∈R 3 and color c∈R 3 .

[0164] Here, the position of the Gaussian sphere corresponding to the reference point can be determined by combining the reference point's location and attributes. Specifically, given a location at x... v The formula for calculating the position of the Gaussian sphere corresponding to the reference point is described as follows: {μ0,...,μ k-1}=x v +{O0,...,O k-1}·l v .

[0165] Through individual multilayer perceptrons (MLPs), features f at reference points can be directly obtained. v The relative viewing distance δ between the camera and the reference point vc and perspective direction Decoding yields the attribute features of k 3D Gaussian spheres. The MLP used to calculate each attribute feature can be denoted as F. α F c F q and F s These are used to decode and obtain opacity, color, quaternion, and scaling features, respectively.

[0166] For example, the opacity value of a 3D Gaussian sphere generated from a reference point can be calculated using the following formula: Other properties of a 3D Gaussian sphere, such as color {c i}、 Quaternion {q i} and scaling features {s i All of these are derived in a similar manner, and will not be repeated here.

[0167] Next, for each reference point, generate k Gaussian spheres, and select m Gaussian spheres that are closest to the reference point as target Gaussian spheres, where m = [k / 3]. The first distances between each target Gaussian sphere and the reference point are d1, d2, d3, ..., d... m The reciprocal of the first distance mentioned above is multiplied by the feature f of the reference point according to the following formula. v Weighted attribute context features f h :

[0168] The process of quantization and entropy coding:

[0169] After obtaining the attribute context features, the entropy of the reference point attribute can be minimized using the attribute context features, thereby reducing the number of bits when encoding the reference point attribute using entropy coding.

[0170] To facilitate entropy coding, the value of the baseline attribute A can be quantized into a finite set. For example, various methods such as traditional rounding and self-quantization can be used.

[0171] In the adaptive quantization method described above, MLPq can be used to estimate the quantization parameters. During the quantization process, the context model MLPq and attribute context features f are utilized. i h The input is used to predict the quantization parameter r. i ∈R 1 The specific formula can be described as r i =MLP q (f i h The quantization parameter is used to adjust the predefined quantization step size Q0. Then, the predefined quantization step size Q0 is adjusted according to this parameter to obtain the target quantization step size. The specific adjustment formula is described below: q i =Q0×(1+Tanh(r) i After obtaining the target quantization step size, the reference point attributes can be quantized according to the target quantization step size to obtain the quantized reference point attributes.

[0172] Next, we can use the context model MLP c The attribute context features are processed to obtain the Gaussian distribution parameters of each reference point attribute. Using the Gaussian distribution parameters, the quantized attribute context features, and the target quantization step size, the probability distribution of the reference point attribute is obtained. Then, entropy encoding is performed on the reference point attribute based on this probability distribution to obtain the encoding result, i.e., the compressed bitstream shown in Figure 2. The specific formula is described below: μ i ,σ i =MLP c (f i h )

[0173] In the embodiments disclosed herein, entropy encoding can employ a deep learning-based encoder or directly use the entropy function as the loss function for optimization and updating. It should be noted that in MLP training, a single loss function or a combination of multiple loss functions can be used, such as the entropy loss function or the rendering reconstruction loss function.

[0174] As described above, the technical solution disclosed herein uses sparse point clouds from 3D digital human signal acquisition as input for representation generation, and employs the Structure for Motion Recovery (SFM) method to generate sparse voxel mesh data of the 3D data. During the reference point generation and attribute prediction process, the center of each voxel in the sparse voxel mesh data is used as the reference point. For each reference point, k Gaussian spheres can be generated, and the attribute characteristics of each Gaussian sphere can be dynamically predicted based on the reference point features and viewpoint position.

[0175] Then, the generated reference point locations and attributes are input into the feature transformation process for processing. Specifically, the attribute context features are obtained by weighting the reference point locations and attributes, combined with the distance between the generated Gaussian sphere and the reference point. The reference point attributes are then quantized through a quantization process to obtain the quantized reference point attributes. Next, the attribute context features output from the feature transformation process are used as context to predict the probability distribution of the reference point attributes, thereby improving the efficiency of the reference point attribute entropy encoding stage.

[0176] This disclosure presents a 3DGS-based digital human compression scheme, which transforms the digital human data compression problem into Gaussian sphere compression. The overall system takes the sparse point cloud of the 3D data of the 3D digital human as input, and through the above process, finally outputs the entropy-encoded bitstream of the reference point position and its attributes of the 3D digital human, thereby reducing the amount of data transmitted over the network.

[0177] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0178] Based on the same inventive concept, this disclosure also provides a 3D digital human and scene processing device corresponding to the 3D digital human and scene processing method. Since the principle of the device in this disclosure for solving the problem is similar to the 3D digital human and scene processing method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0179] This disclosure provides a 3D digital human and scene processing apparatus, comprising:

[0180] The acquisition module is used to acquire 3D data based on 3D Gaussian sputtering 3DGS.

[0181] The processing module is used to perform characterization, quantization, and / or entropy encoding on the 3D data.

[0182] In one possible implementation, the processing module includes:

[0183] The representation generation module is used to generate representations of the 3D data, specifically including: processing the 3D data to obtain the reference point position and reference point attribute of each reference point, and performing feature transformation based on the reference point position and the reference point attribute, combined with the Gaussian sphere position generated corresponding to the reference point, to obtain the attribute context feature of each reference point.

[0184] The quantization module is used to quantize the reference point attributes of the reference point using the attribute context features;

[0185] The entropy encoding module is used to entropy encode the quantized reference point attributes based on the attribute context features to obtain the encoding result.

[0186] Referring to Figure 3, which is a schematic diagram of a 3D digital human and scene processing device provided in an embodiment of this disclosure, the device includes: an acquisition module 10, a characterization generation module 20, a quantization module 30, and an entropy encoding module 40; wherein,

[0187] The acquisition module 10 is used to acquire 3D data based on 3D Gaussian sputtering 3DGS;

[0188] The characterization generation module 20 is used to generate a characterization of the 3D data, specifically including: processing the 3D data to obtain the reference point position and reference point attribute of each reference point, and performing feature transformation based on the reference point position and the reference point attribute, combined with the Gaussian sphere position generated corresponding to the reference point, to obtain the attribute context feature of each reference point.

[0189] Quantization module 30 is used to quantize the reference point attributes of the reference point using the attribute context features;

[0190] Entropy encoding module 40 is used to entropy encode the quantized reference point attribute according to the attribute context features to obtain the encoding result.

[0191] In one possible implementation, the characterization generation module is further configured to:

[0192] The 3D data is processed to generate reference points and predict attributes, thereby obtaining the reference point position and reference point attributes of each reference point.

[0193] In one possible implementation, the characterization generation module is further configured to: convert the 3D data into voxel mesh data;

[0194] Based on the position of the center point of each voxel in the voxel mesh data, the position of the reference point is determined, and the reference point attribute of the reference point corresponding to the position of the reference point is determined.

[0195] In one possible implementation, the characterization generation module is further configured to:

[0196] Based on the location and attributes of the reference point, determine the locations of multiple Gaussian spheres corresponding to the reference point;

[0197] Based on each Gaussian sphere position, the reference point position, and the reference point features, the attribute context features of the reference point are determined.

[0198] In one possible implementation, the characterization generation module is further configured to:

[0199] Based on the position of each Gaussian sphere, a predetermined number of target Gaussian spheres that are closest to the reference point are determined from among the plurality of Gaussian spheres;

[0200] Based on the Gaussian sphere position of the target Gaussian sphere, the reference point position, and the reference point characteristics, the attribute context features of the reference point are determined.

[0201] In one possible implementation, the characterization generation module is further configured to:

[0202] Based on the position of the target Gaussian sphere, the distance between the target Gaussian sphere and the reference point is determined, resulting in multiple first distances;

[0203] Summing the reciprocals of each of the first distances yields the summation result;

[0204] The product between the summation result and the feature of the reference point is calculated to obtain the attribute context feature of the reference point.

[0205] In one possible implementation, the quantization module is further configured to:

[0206] The target quantization step size of the reference point attribute is determined using the attribute context features.

[0207] The reference point attributes are quantized using the target quantization step size to obtain the quantized reference point attributes.

[0208] In one possible implementation, the quantization module is further configured to:

[0209] The attribute context features are processed by a first multilayer perceptron to obtain quantization parameters;

[0210] The target quantization step size is obtained by adjusting the predefined quantization step size using the quantization parameters.

[0211] In one possible implementation, the entropy coding module is further used for:

[0212] The probability distribution of the reference point attribute is determined based on the attribute context features and the quantized reference point attribute.

[0213] The baseline point attributes are entropy-encoded using the probability distribution to obtain the encoding result.

[0214] In one possible implementation, the entropy coding module is further used for:

[0215] The attribute context features are processed by the second multilayer perceptron to obtain the Gaussian distribution parameters of each sub-attribute in the baseline attribute;

[0216] The probability distribution of the reference point attribute is calculated based on the Gaussian distribution parameters, the quantized reference point attribute, and the target quantization step size of the reference point attribute.

[0217] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0218] Corresponding to the 3D digital human and scene processing method in Figure 1, this disclosure also provides an electronic device 400, as shown in Figure 4, which is a schematic diagram of the structure of the electronic device 400 provided in this disclosure, including:

[0219] The system includes a processor 41, a memory 42, and a bus 43. The memory 42 stores execution instructions and includes main memory 421 and external memory 422. The main memory 421, also called internal memory, temporarily stores the computational data in the processor 41, as well as data exchanged with external memory such as a hard disk. The processor 41 exchanges data with the external memory 422 through the main memory 421. When the electronic device 400 is running, the processor 41 communicates with the memory 42 through the bus 43, causing the processor 41 to execute the following instructions:

[0220] Acquire 3D data based on 3D Gaussian sputtering 3DGS;

[0221] The 3D data is characterized, generated, quantized, and / or entropy encoded.

[0222] Optionally, the processor 41 communicates with the memory 42 via a bus 43, causing the processor 41 to execute the following instructions:

[0223] Acquire 3D data based on 3D Gaussian sputtering 3DGS;

[0224] The 3D data is characterized and generated, specifically including: processing the 3D data to obtain the reference point position and reference point attribute of each reference point, and performing feature transformation based on the reference point position and the reference point attribute, combined with the Gaussian sphere position generated corresponding to the reference point, to obtain the attribute context feature of each reference point.

[0225] The reference point attributes of the reference point are quantified using the attribute context features.

[0226] The quantized baseline attribute is entropy encoded based on the attribute context features to obtain the encoding result.

[0227] This disclosure also provides a computer-readable storage medium storing a computer program. When a processor executes the computer program, it performs the steps of the 3D digital human and scene processing method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0228] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the 3D digital human and scene processing method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0229] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0230] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0231] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0232] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0233] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0234] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for processing 3D digital humans and scenes, comprising: Acquire 3D data based on 3D Gaussian sputtering 3DGS; The 3D data is characterized, generated, quantized, and / or entropy encoded.

2. The method according to claim 1, wherein the characterization, quantization, and / or entropy encoding of the 3D data comprises: The 3D data is characterized and generated, specifically including: processing the 3D data to obtain the reference point position and reference point attribute of each reference point, and performing feature transformation based on the reference point position and the reference point attribute, combined with the Gaussian sphere position generated corresponding to the reference point, to obtain the attribute context feature of each reference point. The reference point attributes of the reference point are quantified using the attribute context features. The quantized baseline attribute is entropy encoded based on the attribute context features to obtain the encoding result.

3. The method according to claim 2, wherein, The process of processing the 3D data to obtain the reference point position and reference point attributes of each reference point includes: The 3D data is processed to generate reference points and predict attributes, thereby obtaining the reference point position and reference point attributes of each reference point.

4. The method according to claim 3, wherein, The process of generating reference points and predicting attributes from the 3D data to obtain the reference point position and attributes of each reference point includes: The 3D data is converted into voxel mesh data; Based on the position of the center point of each voxel in the voxel mesh data, the position of the reference point is determined, and the reference point attribute of the reference point corresponding to the position of the reference point is determined.

5. The method according to claim 2, wherein, The feature transformation based on the reference point location and the reference point attributes, combined with the Gaussian sphere location generated corresponding to the reference point, yields the attribute context features of each reference point, including: Based on the location and attributes of the reference point, determine the locations of multiple Gaussian spheres corresponding to the reference point; Based on each Gaussian sphere position, the reference point position, and the reference point features, the attribute context features of the reference point are determined.

6. The method according to claim 5, wherein, The step of determining the positions of multiple Gaussian spheres corresponding to the reference point based on the reference point's position and attributes includes: Calculate the product between the learnable offset in the reference point attribute and the scaling factor in the reference point attribute; The positions of the Gaussian spheres are obtained by summing the product and the reference point position.

7. The method according to claim 5, wherein, The step of determining the attribute context features of the reference point based on the position of each Gaussian sphere, the position of the reference point, and the features of the reference point includes: Based on the position of each Gaussian sphere, a predetermined number of target Gaussian spheres that are closest to the reference point are determined from among the plurality of Gaussian spheres; Based on the Gaussian sphere position of the target Gaussian sphere, the reference point position, and the reference point characteristics, the attribute context features of the reference point are determined.

8. The method according to claim 7, wherein, The step of determining the attribute context features of the reference point based on the Gaussian sphere position of the target Gaussian sphere, the reference point position, and the reference point features includes: Based on the position of the target Gaussian sphere, the distance between the target Gaussian sphere and the reference point is determined, resulting in multiple first distances; Summing the reciprocals of each of the first distances yields the summation result; The product between the summation result and the feature of the reference point is calculated to obtain the attribute context feature of the reference point.

9. The method according to claim 2, wherein, The step of quantifying the reference point attributes of the reference point using the attribute context features includes: The target quantization step size of the reference point attribute is determined using the attribute context features. The reference point attributes are quantized using the target quantization step size to obtain the quantized reference point attributes.

10. The method according to claim 9, wherein, The determination of the target quantization step size for the reference point attribute includes: The attribute context features are processed by a first multilayer perceptron to obtain quantization parameters; The target quantization step size is obtained by adjusting the predefined quantization step size using the quantization parameters.

11. The method according to claim 2, wherein, The step of entropy encoding the quantized reference point attribute based on the attribute context features to obtain the encoding result includes: The probability distribution of the reference point attribute is determined based on the attribute context features and the quantized reference point attribute. The baseline point attributes are entropy-encoded using the probability distribution to obtain the encoding result.

12. The method according to claim 11, wherein, The step of determining the probability distribution of the reference point attribute based on the attribute context features and the quantized reference point attribute includes: The attribute context features are processed by the second multilayer perceptron to obtain the Gaussian distribution parameters of each sub-attribute in the baseline attribute; The probability distribution of the reference point attribute is calculated based on the Gaussian distribution parameters, the quantized reference point attribute, and the target quantization step size of the reference point attribute.

13. A processing device for 3D digital humans and scenes, comprising: The acquisition module is used to acquire 3D data based on 3D Gaussian sputtering 3DGS. The processing module is used to perform characterization, quantization, and / or entropy encoding on the 3D data.

14. The apparatus according to claim 13, wherein, The processing module includes: The representation generation module is used to generate representations of the 3D data, specifically including: processing the 3D data to obtain the reference point position and reference point attribute of each reference point, and performing feature transformation based on the reference point position and the reference point attribute, combined with the Gaussian sphere position generated corresponding to the reference point, to obtain the attribute context feature of each reference point. The quantization module is used to quantize the reference point attributes of the reference point using the attribute context features; The entropy encoding module is used to entropy encode the quantized reference point attributes based on the attribute context features to obtain the encoding result.

15. An electronic device comprising: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the processing method for a 3D digital human and scene as described in any one of claims 1 to 12.

16. A computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the processing method for a 3D digital human and scene as described in any one of claims 1 to 12.

17. A computer program product stored in a storage medium, the program product being executed by at least one processor to implement the steps of the processing method for a 3D digital human and scene as claimed in any one of claims 1 to 12.

Images