Three-dimensional scene reconstruction method and apparatus, and device and storage medium

Abstract

The embodiments of the present application relate to the technical field of computer vision. Provided are a three-dimensional scene reconstruction method and apparatus, and a device and a storage medium. The method comprises: acquiring an anchor-based initial three-dimensional representation model corresponding to a target scene; for each anchor, performing compression encoding on anchor attributes to obtain an encoded bitstream corresponding to the target scene; performing entropy decoding on the encoded bitstream to obtain a decoded feature attribute, a decoded size attribute and a decoded offset attribute; on the basis of the decoded feature attributes and a feature attribute mean value, performing weighted prediction to obtain weighted feature attributes, and performing channel-wise grouping on the weighted feature attributes to obtain rendered feature attributes; and finally, on the basis of the rendered feature attributes, the decoded size attributes and the decoded offset attributes, generating a three-dimensional representation model of the target scene. By means of a weighted prediction process and a channel-wise grouping process, feature attributes of each anchor are simplified, so as to improve the rate distortion performance of an initial three-dimensional representation model while reducing a storage space.

Description

3D scene reconstruction methods, devices, equipment and storage media Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to methods, apparatus, devices and storage media for three-dimensional scene reconstruction. Background Technology

[0002] 3D Gaussian Splatting (3DGS), as a 3D scene representation method, surpasses previous-generation scene representation techniques such as point clouds and Neural Radiation Fields (NeRF) in terms of training speed, rendering speed, and compositing quality. 3DGS represents a 3D scene as a set of Gaussian ellipsoids with local scene properties (such as opacity, color, and size), and uses a fast and differentiable rasterization pipeline to synthesize virtual views. However, 3DGS relies on numerous Gaussian ellipsoids to achieve high-quality view compositing results, leading to a massive amount of data processing, complex computation, and high storage and transmission costs.

[0003] Among related technologies, Scaffold-GS proposes an anchor-based structured 3DGS representation method. This method uses anchor points to cluster Gaussian ellipsoids with similar properties within local regions. It then uses the offset and feature attributes of the anchor points to predict the various properties of the Gaussian points, thereby achieving more efficient rendering. While Scaffold-GS reduces the model size to some extent by clustering Gaussian ellipsoids into anchor point attributes, the storage requirements for the anchor point features remain significant. Summary of the Invention

[0004] The main objective of this application is to propose a three-dimensional scene reconstruction method, apparatus, device, and storage medium to reduce the storage footprint of anchor point features in the anchor point-based structured 3DGS representation process.

[0005] To achieve the above objectives, a first aspect of this application proposes a three-dimensional scene reconstruction method, comprising:

[0006] The target scene is obtained from multiple perspective images, and a sparse point cloud is obtained based on the perspective images. An initial 3D representation model based on anchor points is initialized based on the sparse point cloud. The initial 3D representation model includes multiple anchor points and anchor point attributes corresponding to each anchor point. The anchor point attributes include feature attributes, size attributes, and position offset.

[0007] For each anchor point, the anchor point attributes are compressed and encoded using a preset entropy model to obtain the encoded bitstream corresponding to the target scene;

[0008] Entropy decoding is performed on the encoded bitstream to obtain decoding feature attributes, decoding size attributes, and decoding offset attributes;

[0009] We perform weighted prediction based on the decoded feature attributes and the mean of the feature attributes predicted based on the preset entropy model to obtain weighted feature attributes, and group the weighted feature attributes by channel dimension to obtain rendering feature attributes.

[0010] A 3D representation model of the target scene is generated based on the rendering feature attributes, the decoding size attributes, and the decoding offset attributes.

[0011] In some embodiments, the preset entropy model is a context model, including a binary hash table and a first multilayer perceptron. The step of compressing and encoding the anchor attributes using the preset entropy model to obtain the encoded bitstream corresponding to the target scene includes:

[0012] The context features are obtained by interpolating the coordinates of the anchor points in the binary hash table.

[0013] The context features are input into the first multilayer perceptron for data prediction to obtain probability distribution parameters and quantization step size;

[0014] The anchor point attribute is quantized using the quantization step size to obtain the quantized attribute, and the quantized attribute is entropy encoded using the probability distribution parameter to obtain the encoded bitstream.

[0015] In some embodiments, the entropy decoding of the encoded bitstream to obtain decoding feature attributes, decoding size attributes, and decoding offset attributes includes:

[0016] The encoded bitstream is entropy-decoded using the probability distribution parameters to obtain decoded quantization attributes, which include quantization feature attributes, quantization size attributes, and quantization position offset.

[0017] Obtain the quantization step size corresponding to each of the decoded quantization attributes, multiply the quantization feature attribute and the quantization step size to obtain the decoded feature attribute, multiply the quantization size attribute and the quantization step size to obtain the decoded size attribute, and multiply the quantization position offset and the quantization step size to obtain the decoded offset attribute.

[0018] In some embodiments, the step of performing weighted prediction based on the mean of the feature attributes and the decoded feature attributes to obtain weighted feature attributes includes:

[0019] The context features are input into a second multilayer perceptron for data prediction to obtain feature channel fusion weights.

[0020] Weighted feature attributes are obtained by performing weighted prediction based on the feature channel fusion weights, the mean of the feature attributes, and the decoded feature attributes.

[0021] In some embodiments, the step of inputting the contextual features into a second multilayer perceptron for data prediction to obtain feature channel fusion weights includes:

[0022] The contextual features are input into a second multilayer perceptron for data prediction to obtain a first intermediate value;

[0023] The first intermediate value is input into a preset activation function for data calculation to obtain the second intermediate value;

[0024] The difference between the first and the second intermediate value is obtained to obtain the feature channel fusion weight.

[0025] In some embodiments, the step of performing weighted prediction based on the feature channel fusion weights, the mean of the feature attributes, and the decoded feature attributes to obtain weighted feature attributes includes:

[0026] The product of the feature channel fusion weight and the decoded feature attribute is obtained to get the third intermediate value;

[0027] Obtain the difference between the weights fused with the feature channels to get the intermediate weight value, and multiply the intermediate weight value by the mean of the feature attributes to get the fourth intermediate value;

[0028] The weighted feature attribute is obtained by summing the third intermediate value and the fourth intermediate value.

[0029] In some embodiments, grouping the weighted feature attributes by channel dimension to obtain rendering feature attributes includes:

[0030] Based on a preset grouping method, the weighted feature attributes are divided into first channel features and second channel features in the channel dimension;

[0031] The first channel feature and the second channel feature are added together to obtain the fifth intermediate value, and one of the first channel feature and the second channel feature is selected as the residual feature.

[0032] The rendering feature attribute is obtained by splicing the non-residual feature and the fifth intermediate value.

[0033] To achieve the above objectives, a second aspect of this application provides a three-dimensional scene reconstruction apparatus, comprising:

[0034] Point cloud processing module: used to acquire view images of the target scene from multiple perspectives, and obtain sparse point cloud based on the view images. Initialize an anchor-based 3D representation model based on the sparse point cloud. The initial 3D representation model includes multiple anchor points and anchor point attributes corresponding to each anchor point. The anchor point attributes include feature attributes, size attributes, and position offset.

[0035] Entropy coding module: used to compress and encode the attributes of each anchor point to obtain the encoded bitstream corresponding to the target scene;

[0036] Entropy decoding module: used to perform entropy decoding on the encoded bitstream to obtain decoding feature attributes, decoding size attributes, and decoding offset attributes;

[0037] Redundancy processing module: used to estimate the mean of feature attributes using a preset entropy model, perform weighted prediction based on the decoded feature attributes and the mean of feature attributes to obtain weighted feature attributes, and group the weighted feature attributes by channel dimension to obtain rendering feature attributes;

[0038] Rendering and Reconstruction Module: Used to generate a 3D representation model of the target scene based on the rendering feature attributes, the decoding size attributes, and the decoding offset attributes.

[0039] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0040] To achieve the above objectives, a fourth aspect of the present application provides a storage medium that stores a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0041] The 3D scene reconstruction method, apparatus, device, and storage medium proposed in this application acquire viewpoint images of the target scene from multiple perspectives, obtain sparse point clouds based on the viewpoint images, and initialize an initial 3D representation model based on anchor points according to the sparse point clouds. The initial 3D representation model includes multiple anchor points and anchor point attributes for each anchor point. Anchor point attributes include feature attributes, size attributes, and position offsets. For each anchor point, the anchor point attributes are compressed and encoded to obtain an encoded bitstream corresponding to the target scene. Then, the encoded bitstream is entropy-decoded to obtain decoded feature attributes, decoded size attributes, and decoded offset attributes. Next, weighted prediction is performed based on the decoded feature attributes and the mean of feature attributes predicted using a preset model to obtain weighted feature attributes. Finally, the weighted feature attributes are grouped by channel dimension to obtain rendering feature attributes. Finally, a 3D representation model of the target scene is generated based on the rendering feature attributes, decoded size attributes, and decoded offset attributes. In this application embodiment, the feature attributes of the anchor points are reasonably simplified. On the one hand, although the anchor point's feature attributes contain rich scene information, some of this information overlaps with the mean feature attributes predicted by the preset entropy model. Therefore, by introducing the mean feature attributes through a weighted prediction process, the anchor point's feature attributes only need to represent the missing or inaccurate scene information in the mean feature attributes, thus effectively reducing the scene information contained in the feature attributes. On the other hand, by analyzing the similarity between channels, it was found that the similarity between all channels is very high. Therefore, if divided into two groups, the similarity between the two groups will also be very high. Thus, residuals with low storage cost are used to model the subtle differences between the two groups. In other words, related technologies require storing two sets of features, while this application embodiment only requires storing one set of features and residuals. The other set of features can be represented by the stored set of features combined with the residuals. This reduces channel redundancy of feature attributes while retaining key channel information, further reducing the storage footprint of the anchor point's feature attributes. By optimizing the anchor point attributes, it is possible to reduce transmission bandwidth, reduce storage space, and improve the rate-distortion performance of the 3D representation model while obtaining higher quality reconstruction results. Attached Figure Description

[0042] Figure 1 is a flowchart of the three-dimensional scene reconstruction method provided in the embodiment of this application.

[0043] Figure 2 is a flowchart of the process of using a preset entropy model to compress and encode anchor point attributes to obtain the encoded bitstream corresponding to the target scene, according to an embodiment of this application.

[0044] Figure 3 is a flowchart of entropy decoding of the encoded bitstream to obtain decoding feature attributes, decoding size attributes and decoding offset attributes provided in an embodiment of this application.

[0045] Figure 4 is a schematic diagram of the redundancy analysis of the feature attributes provided in the embodiments of this application.

[0046] Figure 5 is a flowchart of a process for obtaining weighted feature attributes by performing weighted prediction based on the mean of feature attributes and decoded feature attributes, according to an embodiment of this application.

[0047] Figure 6 is a flowchart of the process of inputting contextual features into a second multilayer perceptron for data prediction and obtaining feature channel fusion weights, as provided in an embodiment of this application.

[0048] Figure 7 is a flowchart of a process provided in this application to obtain weighted feature attributes by performing weighted prediction based on feature channel fusion weights, feature attribute mean and decoded feature attributes.

[0049] Figure 8 is a schematic diagram of channel correlation analysis of feature attributes provided in the embodiments of this application.

[0050] Figure 9 is a flowchart of the process of grouping weighted feature attributes by channel dimension to obtain rendered feature attributes, provided in an embodiment of this application.

[0051] Figure 10 is a schematic diagram of the overall process provided in the embodiments of this application.

[0052] Figure 11 is a structural block diagram of a three-dimensional scene reconstruction device provided in another embodiment of this application.

[0053] Figure 12 is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that although functional modules are grouped in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module grouping in the device or the order in the flowchart.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0057] First, let's analyze some of the terms used in this application:

[0058] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0059] With the rapid development of technologies such as virtual reality (VR), augmented reality (AR), and digital twins, 3D scene reconstruction technology has become an important research direction. Traditional 3D reconstruction methods mostly rely on depth sensors (such as LiDAR) to extract scene geometric information, which usually suffers from high cost, high computational complexity, and low reconstruction accuracy. In recent years, to achieve more efficient and accurate 3D reconstruction, methods based on deep learning and viewpoint synthesis have attracted much attention in academia and industry.

[0060] 3DGaussian Splatting (3DGS), as a 3D scene representation method, surpasses previous-generation scene representation techniques such as point clouds and Neural Radiation Fields (NeRF) in terms of training speed, rendering speed, and synthesis quality. 3DGS represents a 3D scene as a set of Gaussian ellipsoids with local scene properties (such as opacity, color, and size), and synthesizes virtual views using a fast and differentiable rasterization pipeline. However, 3DGS relies on numerous Gaussian ellipsoids to achieve high-quality view synthesis results, leading to a massive amount of data processing, complex computation, and high costs for model storage and transmission.

[0061] Among related technologies, Scaffold-GS proposes an anchor-based structured 3DGS representation method. This method uses anchor points to cluster Gaussian ellipsoids with similar properties within local regions. It then uses the offset and feature attributes of the anchor points to predict the various properties of the Gaussian points, thereby achieving more efficient rendering. While Scaffold-GS reduces the model size to some extent by clustering Gaussian ellipsoids into anchor point attributes, the storage requirements for the anchor point features remain significant.

[0062] Based on this, embodiments of this application provide a 3D scene reconstruction method, apparatus, device, and storage medium that reasonably simplifies the feature attributes of anchor points. On one hand, although the feature attributes of anchor points contain rich scene information, some information overlaps with the mean feature attribute predicted by a preset entropy model. Therefore, by introducing the mean feature attribute through a weighted prediction process, the feature attributes of anchor points only need to represent the missing or inaccurate scene information in the mean feature attribute, thus effectively reducing the scene information contained in the feature attributes. On the other hand, by analyzing the similarity between channels, it is found that the similarity between all channels is very high. Therefore, if divided into two groups, the similarity between the two groups will also be very high. Thus, residuals with low storage cost are used to model the subtle differences between the two groups. In other words, related technologies require storing two sets of features, while embodiments of this application only require storing one set of features and residuals. The other set of features is represented by the stored set of features combined with the residuals. This reduces channel redundancy of feature attributes while retaining key channel information, further reducing the storage footprint of the anchor point's feature attributes. By optimizing anchor point attributes, it is possible to reduce transmission bandwidth, decrease storage space, and improve rate-distortion performance of 3D representation models while obtaining higher quality reconstruction results.

[0063] This application provides a three-dimensional scene reconstruction method, apparatus, device, and storage medium, which are specifically described through the following embodiments. First, the three-dimensional scene reconstruction method in this application embodiment is described.

[0064] This application's embodiments can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0065] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0066] The 3D scene reconstruction method provided in this application relates to the field of computer vision technology. This method can be applied to a terminal, a server, or a computer program running on either a terminal or a server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client that supports 3D scene generation, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. The 3D scene reconstruction method can be executed by the terminal or the server, or by the terminal and the server working together.

[0067] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. Additionally, the terminal can also be a smart in-vehicle device. This smart in-vehicle device uses the 3D scene reconstruction method of this embodiment to provide related services and enhance the driving experience. The server can be an independent server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The terminal and server can connect via Bluetooth, Universal Serial Bus (USB), or network communication methods; this embodiment does not impose any limitations.

[0068] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0069] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0070] The following describes the three-dimensional scene reconstruction method in the embodiments of this application.

[0071] Figure 1 is an optional flowchart of a three-dimensional scene reconstruction method provided in an embodiment of this application. The method in Figure 1 may include, but is not limited to, steps 110 to 150. It is also understood that this embodiment does not specifically limit the order of steps 110 to 150 in Figure 1, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0072] Step 110: Obtain view images of the target scene from multiple perspectives, and obtain sparse point clouds based on the view images. Initialize an initial 3D representation model based on anchor points based on the sparse point clouds.

[0073] In one embodiment, the target scene is the scene that requires 3D modeling. For this target scene, a 360-degree panoramic surround layout of image acquisition devices can be deployed to achieve synchronous acquisition. This surround layout can comprehensively cover the target scene, ensuring no angular information is missed, thus enabling the acquisition of data from all directions. Different image acquisition devices correspond to different viewpoints, similar to observing the target scene from different positions, where the view differs. Image acquisition devices are used to acquire corresponding viewpoint images at each viewpoint.

[0074] In one embodiment, the Colmap calibration method is used to perform camera calibration on all viewpoint images acquired at the same time to obtain a sparse point cloud characterizing the target scene, as well as the intrinsic and extrinsic parameters corresponding to each image acquisition device.

[0075] Specifically, this embodiment uses the Colmap calibration method to analyze the feature information in each viewpoint image. This feature information includes corner points, edges, and other significant visual features in the viewpoint image. Then, using a specific feature extraction algorithm, these feature points are accurately identified. Next, a matching operation is performed based on the correspondence between these feature points in different viewpoint images. For example, for a corner point of a building in one viewpoint image, the corresponding corner point of the same building needs to be found in other viewpoint images. During the matching process, factors such as the position of the feature point and the grayscale changes of surrounding pixels are comprehensively considered to ensure the accuracy of the matching.

[0076] After feature matching is completed, the intrinsic and extrinsic parameters corresponding to each image acquisition device are calculated. The intrinsic parameters reflect the imaging characteristics of the camera itself, such as the camera's focal length, principal point position, and possible lens distortion parameters; the extrinsic parameters reflect the camera's position and attitude information in space, such as the camera's rotation angle and translation vector relative to the target scene.

[0077] Simultaneously, using the matched feature point information, a sparse point cloud representing the target scene is obtained. This sparse point cloud is a discretized representation of the target scene in three-dimensional space, composed of numerous feature points. These feature points are relatively sparsely distributed and outline the approximate contours of the main objects in the target scene, as well as key features such as their spatial relationships.

[0078] In one embodiment, in the Scaffold-GS scenario, the obtained sparse point cloud is initialized to obtain an initial 3D representation model based on anchor points. This initial 3D representation model consists of multiple different 3D Gaussian distributions, each of which can be regarded as a 3D ellipsoid. Multiple 3D ellipsoids are used to fit the sparse point cloud. Anchor points are reference points used to organize and process Gaussian ellipsoids, similar to a "center" or "reference point," playing a role in grouping and managing Gaussian ellipsoids within a local area. In the initial 3D representation model, each anchor point contains one or more 3D Gaussian distributions, and a 3D Gaussian distribution can be represented by the Gaussian point at its center.

[0079] In the case of a single three-dimensional Gaussian distribution, the distribution can typically describe the relatively simple and concentrated data distribution near the anchor point based on its specific distribution characteristics. For example, for an anchor point corresponding to a relatively isolated and regularly shaped small object, a single three-dimensional Gaussian distribution is sufficient to reflect its spatial location and the density of data within a certain range.

[0080] When an anchor point corresponds to more than one 3D Gaussian distribution, it means that the data distribution in the area where the anchor point is located is relatively complex, requiring the use of multiple 3D Gaussian distributions to jointly characterize it. For example, in a scenario involving complex mechanical structures, a key node of a large mechanical device can be considered as an anchor point. The surrounding components have different shapes and complex spatial layouts. In this case, multiple 3D Gaussian distributions are needed, each with different Gaussian parameters such as center position and covariance. Through the cooperation and synergy of these Gaussian points, the complex spatial morphology around the anchor point, the density variations of the data, and the relationships between different components can be accurately presented.

[0081] In one embodiment, each anchor point also has a corresponding anchorage attribute a. Specifically, the anchor point attributes include feature attribute f, size attribute l, and position offset o, denoted as a∈{f, l, o}.

[0082] Among them, the feature attribute f reflects some inherent characteristics of the anchor point, such as content related to the color and geometry of objects in the target scene. For example, in a 3D scene containing multiple objects, different anchor points often have different feature attributes.

[0083] The size attribute *l* is related to the size of the initial 3D representation model associated with the anchor point. In 3D space, the initial 3D representation model can be used to represent the shape and size of an object. For larger objects, the size attribute value of the corresponding anchor point may be relatively large; while for smaller objects, the size attribute value is relatively small.

[0084] Furthermore, since each anchor point contains one or more 3D Gaussian distributions, the position offset attribute 'o' represents the positional offset of the 3D Gaussian distributions within the anchor point. Using the position offset, the positional relationship between each 3D Gaussian distribution and the anchor point can be clearly defined. Therefore, the coordinates of the center position of each 3D Gaussian distribution can be uniquely determined using the anchor point's coordinates and the position offset. Once the anchor point's coordinates and related attribute information are determined, an initial 3D representation model based on the anchor point can be constructed. The 3D Gaussian distributions around each anchor point can be used to describe the probability distribution of the scene in that local region.

[0085] Step 120: For each anchor point, use a preset entropy model to compress and encode the anchor point attributes to obtain the encoded bitstream corresponding to the target scene.

[0086] In one embodiment, a preset entropy model is used to compress anchor attributes in a 3D scene representation model. This preset entropy model can be a context model, a priori model, etc. For example, the preset entropy model can be a context model, which includes a binary hash table and a first multilayer perceptron. Referring to Figure 2, Figure 2 is a flowchart of a process provided in this application to compress and encode anchor attributes using a preset entropy model to obtain an encoded bitstream corresponding to the target scene, including:

[0087] Step 210: Use the coordinates of the anchor point to perform interpolation in the binary hash table to obtain the context features.

[0088] In some embodiments, to achieve optimal rate-distortion performance, a binary hash table is pre-constructed to store the context information corresponding to each anchor point. This includes, for example, the distribution of objects around the anchor point in the target scene, its spatial relationship with adjacent feature points, and any semantic information that may be involved. By pre-constructing this binary hash table, this rich context information can be stored in a structured form, facilitating subsequent retrieval and use. Furthermore, this binary hash table can be optimized during training. During training, based on a large number of data samples and corresponding annotation information, the method of storing information, association rules, and related parameters in the hash table are continuously adjusted. For example, when processing data containing various types of scenes (such as indoor scenes, outdoor scenes, etc.), by exposing the hash table to various anchor points and their actual contexts, it learns how to more accurately store and reflect context information in different scenes, thereby improving the accuracy and effectiveness of subsequent information retrieval.

[0089] Next, the coordinates of the anchor point in three-dimensional space are input into a binary hash table. By mapping the coordinates, the context feature f corresponding to the anchor point can be obtained. cThe coordinates of the anchor point determine its specific location in the three-dimensional space of the target scene. Using this location information as input is like providing an "index" for the binary hash table, thus clarifying the information to be searched. Since the context information stored in the hash table may be discrete, interpolation operations are used to perform numerical calculations based on the existing stored data in the hash table, and based on the surrounding stored relevant information, to obtain the accurate context features corresponding to the anchor point.

[0090] Step 220: Input the context features into the first multilayer perceptron for data prediction to obtain the probability distribution parameters and quantization step size.

[0091] In one embodiment, the acquired contextual features are input into a first multilayer perceptron to perform data prediction on each attribute of the anchor point, obtaining the quantization step size and the probability distribution parameters corresponding to each attribute. The probability distribution parameters include the attribute mean and the attribute standard deviation. The attribute mean reflects the average level of the anchor point attribute across the whole, while the attribute standard deviation reflects the dispersion of the attribute data relative to the mean, i.e., the data fluctuation.

[0092] The above process can be represented as follows:

[0093] μ a , σ a q a =MLP1(f c ),

[0094] Where, μ a σ represents the mean of the attribute corresponding to attribute a. a q represents the standard deviation of attribute a. a This represents the quantization step size corresponding to attribute a, and MLP1 represents the first multilayer perceptron.

[0095] As can be seen from the above, the mean μ of the feature attribute corresponding to the feature attribute can be obtained through the first multilayer perceptron. f σ, the standard deviation of characteristic attributes f and the quantization step size q of the feature attributes f The mean value μ of the dimension attribute corresponding to the dimension attribute. l Dimensional attribute standard deviation σ l And the quantization step size q of the size attribute l The mean value of the position offset μ corresponding to the position offset o Standard deviation of position offset σ o And the quantization step size of the position offset, position offset q o .

[0096] Step 230: Quantize the anchor point attribute using the quantization step size to obtain the quantized attribute, and entropy encode the quantized attribute using the probability distribution parameter to obtain the encoded bitstream.

[0097] In one embodiment, for each different anchor attribute, it can first be quantized using a quantization step size to obtain the corresponding quantized attribute. Then, the quantized attribute related to the anchor attribute is entropy encoded using the corresponding attribute mean and attribute standard deviation to obtain the encoded bitstream. The process of quantizing the anchor attribute using a quantization step size is represented as follows:

[0098] Where 'a' represents the specific value corresponding to the anchor attribute 'a'.

[0099] In one embodiment, entropy coding employs various encoding methods, such as arithmetic coding and Huffman tree coding. This application does not limit the specific entropy coding method. Taking arithmetic coding as an example, arithmetic coding, as a probability-based entropy coding technique, treats the entire anchor attribute data as a single symbol sequence. When performing arithmetic coding on the anchor attribute, the probability distribution corresponding to each parameter and its value in the anchor attribute is first analyzed. Subsequently, based on this probability information, all data in the anchor attribute is gradually mapped to the corresponding coding interval. By continuously subdividing and determining the coding interval, a unique corresponding encoded bitstream is finally generated. After obtaining the encoded bitstream, subsequent decoding operations are performed.

[0100] In one embodiment, the attribute mean μ corresponding to the probability distribution parameter a and attribute standard deviation σ a It can also be used to reflect the bit rate loss during the encoding process. For example, when the bit rate loss is proportional to the calculated information entropy, the bit rate loss value can be obtained based on the information entropy. Then, the total bit rate loss value can be calculated based on the bit rate loss values ​​of all attributes. Based on this total bit rate loss value, the overall parameters such as the attribute values ​​of the anchor point and the preset entropy model can be adjusted, thereby controlling the bit consumption generated by the attribute-related information of the anchor point during the compression process.

[0101] It is understood that the encoding process in this application embodiment has two advantages. On the one hand, it can compress the storage space of anchor attributes. After encoding, the anchor attributes exist in a more compact encoded bitstream, which greatly saves storage resources. This is very beneficial for storing a large amount of anchor attribute data for a long time or storing related data on devices with limited storage resources. On the other hand, if there is a transmission requirement, given the relatively small amount of data in the encoded bitstream, the transmission rate can also be improved, ensuring that the data can arrive at the receiving end quickly and stably from the sending end.

[0102] In one embodiment, the encoding and corresponding decoding processes are performed within the same processing device. If the processing device has limited resources, performing encoding and decoding operations within this device reduces the amount of data to be processed, alleviating the computational burden on the device, allowing it to complete subsequent processing tasks with limited resources. Furthermore, in applications with high real-time requirements, performing encoding and decoding operations within the same device can also reduce the time spent on intermediate steps such as data transmission, quickly achieving the conversion from raw data to the final reconstructed result, thereby ensuring processing efficiency.

[0103] Furthermore, the encoding and corresponding decoding processes can be performed in a distributed manner. First, the encoding operation is performed at the encoding end, followed by the transmission of the corresponding encoded bitstream, and then the decoding operation is performed on the corresponding processing device at the decoding end. When transmission bandwidth is limited, transmitting the encoded bitstream can effectively reduce the amount of data transmitted and improve transmission efficiency. In this way, the entire reconstruction process will not be affected by delays in the transmission stage, thereby improving the overall reconstruction efficiency. It is understood that this embodiment does not limit the execution location of the encoding and corresponding decoding processes.

[0104] Step 130: Perform entropy decoding on the encoded bitstream to obtain the decoding feature attributes, decoding size attributes, and decoding offset attributes.

[0105] In one embodiment, referring to FIG3, FIG3 is a flowchart of entropy decoding of an encoded bitstream to obtain decoding feature attributes, decoding size attributes, and decoding offset attributes provided by an embodiment of this application, specifically including the following steps:

[0106] Step 310: Use the probability distribution parameters to perform entropy decoding on the encoded bitstream to obtain the decoded quantization attributes.

[0107] In one embodiment, the encoded bitstream is entropy decoded using an entropy decoding method corresponding to entropy encoding to obtain decoded quantization attributes, wherein the decoded quantization attributes include quantization feature attributes, quantization size attributes, and quantization position offset.

[0108] Step 320: Obtain the quantization step size corresponding to each decoded quantization attribute, multiply the quantization feature attribute and the quantization step size to obtain the decoded feature attribute, multiply the quantization size attribute and the quantization step size to obtain the decoded size attribute, and multiply the quantization position offset and the quantization step size to obtain the decoded offset attribute.

[0109] In one embodiment, the quantization step size can be obtained from the preceding calculations, therefore the decoded feature attributes are represented as follows:

[0110] The decoding size attribute is represented as:

[0111] The decoding offset attribute is represented as:

[0112] This yields the decoded attribute information.

[0113] In one embodiment, the decoded attribute information differs between the training and inference processes. During training, mean noise is added to the attributes to simulate the decoding steps, as shown below:

[0114] in, This represents the attribute information of the decoded encoded attribute a' during the training process. Let q represent a uniform distribution, a′ represent the value of the encoded attribute a′ during training, and q represent the uniform distribution. a ′ represents the quantization step size of the encoded attribute a' during training.

[0115] Step 140: Perform weighted prediction based on the decoded feature attributes and the mean of the feature attributes predicted based on the preset entropy model to obtain weighted feature attributes, and group the weighted feature attributes by channel dimension to obtain rendering feature attributes.

[0116] In one embodiment, this application selects feature attributes for redundancy analysis. Unlike related technologies that directly utilize decoded feature attributes obtained from decoding for subsequent rendering and reconstruction processes, this embodiment simplifies and compresses them, thereby reducing the storage requirements of anchor points. Although the feature attributes of anchor points contain rich scene information, some information overlaps with the mean of feature attributes predicted by the preset entropy model. Therefore, by introducing the mean of feature attributes through a weighted prediction process, the feature attributes of anchor points only need to represent the missing or inaccurate scene information in the mean of feature attributes, thus effectively reducing the scene information contained in the feature attributes.

[0117] In one embodiment, referring to Figure 4, which is a schematic diagram of feature attribute redundancy analysis provided in this application embodiment, it can be seen that the regular rendered image obtained from the complete anchor point attributes and the mean rendered image corresponding to the mean of the feature attribute have high similarity. This shows that using the mean of the feature attribute to replace the feature attributes in the anchor point in the rendering process can also achieve good scene geometry and color. Related technologies focus on the role of the mean of the feature attribute in calculating bitrate loss, while ignoring the fact that the mean of the feature attribute carries rich scene information that can be used to assist reconstruction. Therefore, this application embodiment combines the mean of the feature attribute for weighted prediction to reduce data redundancy.

[0118] In one embodiment, referring to FIG5, FIG5 is a flowchart of a weighted prediction based on the mean of feature attributes and decoded feature attributes to obtain weighted feature attributes, which specifically includes the following steps:

[0119] Step 510: Input the context features into the second multilayer perceptron for data prediction to obtain the feature channel fusion weights.

[0120] In one embodiment, the feature channel fusion weight is used to assign corresponding weights to the reconstruction results according to the importance of different feature attributes, derived from the adaptively aggregated feature attribute mean μ. f and decoding feature attributes The guided model uses scene information from the mean to reconstruct the data, reducing the need to decode feature attributes. The required amount of information to be represented. Referring to Figure 6, which is a flowchart of an embodiment of this application, the process of inputting contextual features into a second multilayer perceptron for data prediction to obtain feature channel fusion weights includes the following steps:

[0121] Step 610: Input the contextual features into the second multilayer perceptron for data prediction to obtain the first intermediate value.

[0122] In one embodiment, the first intermediate value is represented as: MLP w (f c )

[0123] Among them, MLP w This represents the second-layer perceptron.

[0124] Step 620: Input the first intermediate value into the preset activation function to perform data calculation and obtain the second intermediate value.

[0125] In one embodiment, if the preset activation function is the sigmoid activation function, then the second intermediate value is represented as: Sigmoid(MLP) w (f c ))

[0126] Step 630: Obtain the difference between the first and second intermediate values ​​to get the feature channel fusion weights.

[0127] In one embodiment, the feature channel fusion weight w is expressed as: w = 1 - Sigmoid(MLP) w (f c ))

[0128] Step 520: Perform weighted prediction based on feature channel fusion weights, feature attribute mean, and decoded feature attributes to obtain weighted feature attributes.

[0129] In one embodiment, referring to Figure 7, which is a flowchart of a weighted prediction based on feature channel fusion weights, mean feature attributes, and decoded feature attributes provided in this application embodiment, the process includes the following steps:

[0130] Step 710: Obtain the product of the feature channel fusion weight and the decoded feature attribute to get the third intermediate value.

[0131] In one embodiment, the third intermediate value is represented as:

[0132] Step 720: Obtain the difference between the weights fused with the feature channels to get the intermediate weight value. Multiply the intermediate weight value by the mean of the feature attributes to get the fourth intermediate value.

[0133] In one embodiment, the fourth intermediate value is represented as: (1-w)·μ f

[0134] Where (1-w) represents the median weight.

[0135] Step 730: Accumulate the third and fourth intermediate values ​​to obtain the weighted feature attributes.

[0136] In one embodiment, weighted feature attributes Represented as:

[0137] In one embodiment, after obtaining the weighted feature attributes, the similarity between channels is analyzed. It is found that the similarity between all channels is very high. Therefore, if they are divided into two groups, the similarity between the two groups will also be very high. Thus, residuals with low storage cost are used to model the subtle differences between the two groups. That is, related technologies require storing two sets of features, while this embodiment only needs to store one set of features and residuals. The other set of features can be represented by the stored set of features combined with the residuals. This can reduce the channel redundancy of feature attributes while retaining key channel information, further reducing the storage footprint of the anchor's feature attributes. Referring to Figure 8, Figure 8 is a schematic diagram of channel correlation analysis of feature attributes provided by this embodiment. The figure uses four channels as an example for illustration. By visualizing the channels and rendering them using different channels, channel rendering image 1, channel rendering image 2, channel rendering image 3, and channel rendering image 4 are obtained respectively. It can be seen that the similarity of these four rendering images is extremely high. This shows that there is a high degree of similarity between the channels of the anchor's feature attributes. Therefore, this embodiment uses channel-dimensional grouping to remove channel redundancy.

[0138] In one embodiment, referring to Figure 9, which is a flowchart of grouping weighted feature attributes by channel dimension to obtain rendered feature attributes according to an embodiment of this application, the process specifically includes the following steps:

[0139] Step 910: Based on the preset grouping method, the weighted feature attributes are divided into first channel features and second channel features in the channel dimension.

[0140] In one embodiment, the preset grouping method can be uniform grouping, i.e., random halving, or other grouping methods; this embodiment is not limited to this. Assuming the channel dimension is D, the preset grouping methods include, but are not limited to:

[0141] 1) Group [0, 1, ..., D / 2-1] as the first group, and [D / 2, D / 2+1, ..., D-1] as the second group; 2) Group [0, 2, ..., D] as the first group, and [1, 3, ..., D-1] as the second group, etc. Based on the preset grouping method, the weighted feature attributes are divided into first channel features and second channel features along the channel dimension, represented as:

[0142] Where D represents the channel dimension, Indicates the characteristics of the first channel. This indicates the characteristics of the second channel.

[0143] Step 920: Add the first channel feature and the second channel feature to obtain the fifth intermediate value, and select one of the first channel feature and the second channel feature as the residual feature.

[0144] In one embodiment, the fifth intermediate value is represented as:

[0145] To reduce redundancy between these two sets of features—specifically, redundancy in the anchor feature channel dimension—it's necessary to select one of the first and second channel features as the residual feature. This reduces the amount of information without compromising the representational power of the anchor feature. Since the rendering results of different channels are highly similar, this selection can be random.

[0146] Step 930: Concatenate the non-residual features and the fifth intermediate value to obtain the rendered feature attributes.

[0147] In one embodiment, if the second channel feature is selected as... Residual features, where non-residual features are the first channel features. This residual prediction-based approach can explicitly utilize the correlation between feature channels, reduce redundancy between the first and second channel features, and reduce the amount of information that needs to be represented in the second channel features, thereby reducing computational overhead.

[0148] The rendering feature attributes are obtained by concatenating the non-residual features and the fifth intermediate value. Represented as:

[0149] Next, utilize rendering feature attributes Replace the decoded feature attributes obtained from the original decoding process and participate in the subsequent rendering process.

[0150] Step 150: Generate a 3D representation model of the target scene based on rendering feature attributes, decoding size attributes, and decoding offset attributes.

[0151] In one embodiment, the rendering feature attributes, decoded size attributes, and decoded offset attributes obtained according to the above process constitute a new 3D representation model, which is used as reconstruction input data. The decoded size attributes and decoded offset attributes are the decoded attribute data, and the reconstruction input data is represented as follows:

[0152] Next, similar to the Scaffold-GS method, the decoded feature attributes, size attributes, and position offsets corresponding to each anchor point are first obtained from the reconstructed input data. Then, the Gaussian distribution attributes corresponding to all 3D Gaussian distributions contained in the anchor point are predicted. For example, the feature attributes are used to predict the relevant categories, materials, and approximate shapes of the 3D Gaussian distributions; the size attributes are used to determine the spatial distribution range of the 3D Gaussian distributions; and the position offsets are used to accurately determine the position of the center of the 3D Gaussian distribution relative to the anchor point.

[0153] After acquiring the Gaussian distribution attributes, the view composition stage begins. In this stage, by combining the intrinsic and extrinsic parameters of the image acquisition device with the Gaussian distribution attributes, the appearance of the target scene from different viewpoints can be determined. Since the Gaussian distribution attributes encompass information such as material, spatial distribution, and relative position, these attributes work together when the target scene is observed from different perspectives, acting like individual "building blocks." These blocks combine according to their characteristics and positional relationships to present the specific appearance of the scene from different viewpoints, ultimately achieving a 3D reconstruction of the target scene.

[0154] In one embodiment, referring to FIG10, FIG10 is a schematic diagram of the overall process provided in an embodiment of the present application. In FIG10, an initial three-dimensional representation model corresponding to multiple anchor points is first generated based on the sparse point cloud. Each anchor point corresponds to a voxel, and each voxel contains one or more Gaussian distributions. In addition to coordinate values, the anchor point also includes feature attribute f, size attribute l, and position offset o.

[0155] Next, we proceed to the entropy modeling process, which includes encoding and decoding. We use the coordinates of the anchor points to perform interpolation operations on a binary hash table to obtain the contextual features f. c , context features f c Inputting the first multilayer perceptron (MLP1), data prediction is performed on each anchor attribute to obtain the attribute mean μ for each attribute. a Attribute standard deviation σ a and the corresponding quantization step size q a .

[0156] Then, based on the obtained mean value μ of the feature attributes f and decoding feature attributes The process of performing mean-based weighted prediction first involves considering the context features f. c Input to the second multilayer perceptron (MLP) w Data prediction is performed to obtain the feature channel fusion weight w. Then, based on the feature channel fusion weight w and the mean of the feature attribute μ... f and decoding feature attributes Perform weighted prediction to obtain weighted feature attributes.

[0157] Next, the channel-dimensional grouping process begins. Based on a preset grouping method, the weighted feature attributes are divided into first-channel features and second-channel features along the channel dimension. Then, the first-channel features and second-channel features are added together to obtain a fifth intermediate value. One of the first-channel features and second-channel features is selected as the residual feature. The non-residual feature and the fifth intermediate value are then concatenated to obtain the rendering feature attributes.

[0158] Finally, the rendering process begins. Using 3D Gaussian splashing technology, Gaussian point attributes corresponding to anchor points are generated based on rendering feature attributes, decoding size attributes, and decoding offset attributes. A 3D representation model of the target scene is then generated based on all Gaussian point attributes.

[0159] The 3D scene reconstruction method provided in this application firstly utilizes a mean-based weighted prediction method to adaptively use the mean of the probability distribution predicted by the entropy model for anchor point features for reconstruction. This reduces the amount of scene information that needs to be represented in the anchor point features, thereby reducing storage overhead without compromising the reconstruction quality of the scene representation model. Furthermore, a cross-channel residual prediction process is performed, dividing the anchor point features into two groups according to the channel dimension. One group is considered as the residual of the other group, fully utilizing the similarity between the anchor point feature attribute channels to reduce redundancy between the two groups of features, thereby reducing the storage requirements of the 3D scene representation model.

[0160] In one embodiment, to verify rendering performance, this application compares and analyzes the 3D static scene compression methods in related technologies using the BungeeNeRF and Mip-NeRF 360 datasets. The selected 3D static scene compression methods include 3DGS, Scaffold-GS, and HAC (Hash-grid Assisted). The selected analysis metrics are: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), and Storage (MB). The specific comparative analysis results are presented in Tables 1 and 2 below.

[0161] The results show that, compared to HAC, the best-performing technology in the relevant field, the embodiments of this application can reduce model storage costs while basically not compromising the rendering quality of the model, and even achieve better rendering quality on the BungeeNeRF dataset. This fully demonstrates the superiority of the 3D scene reconstruction method involved in the embodiments of this application.

[0162] Table 1 shows the analysis results on the BungeeNeRF dataset.

[0163] Table 2 shows the analysis results on the Mip-NeRF 360 dataset.

[0164] The technical solution provided in this application involves acquiring viewpoint images of a target scene from multiple perspectives, obtaining sparse point clouds based on these viewpoint images, and initializing an anchor-based 3D representation model using the sparse point clouds. The initial 3D representation model includes multiple anchor points and anchor point attributes for each anchor point. Anchor point attributes include feature attributes, size attributes, and position offsets. For each anchor point, the anchor point attributes are compressed and encoded to obtain an encoded bitstream corresponding to the target scene. Then, the encoded bitstream is entropy-decoded to obtain decoded feature attributes, decoded size attributes, and decoded offset attributes. Next, weighted prediction is performed based on the decoded feature attributes and the mean of feature attributes predicted using a preset model to obtain weighted feature attributes. Finally, the weighted feature attributes are grouped by channel dimension to obtain rendering feature attributes. Finally, a 3D representation model of the target scene is generated based on the rendering feature attributes, decoded size attributes, and decoded offset attributes. In this application embodiment, the feature attributes of the anchor points are reasonably simplified. On the one hand, although the anchor point's feature attributes contain rich scene information, some of this information overlaps with the mean feature attributes predicted by the preset entropy model. Therefore, by introducing the mean feature attributes through a weighted prediction process, the anchor point's feature attributes only need to represent the missing or inaccurate scene information in the mean feature attributes, thus effectively reducing the scene information contained in the feature attributes. On the other hand, by analyzing the similarity between channels, it was found that the similarity between all channels is very high. Therefore, if divided into two groups, the similarity between the two groups will also be very high. Thus, residuals with low storage cost are used to model the subtle differences between the two groups. In other words, related technologies require storing two sets of features, while this application embodiment only requires storing one set of features and residuals. The other set of features can be represented by the stored set of features combined with the residuals. This reduces channel redundancy of feature attributes while retaining key channel information, further reducing the storage footprint of the anchor point's feature attributes. By optimizing the anchor point attributes, it is possible to reduce transmission bandwidth, reduce storage space, and improve the rate-distortion performance of the 3D representation model while obtaining higher quality reconstruction results.

[0165] This application embodiment also provides a three-dimensional scene reconstruction apparatus, which can implement the above-described three-dimensional scene reconstruction method. Referring to FIG11, the apparatus includes:

[0166] Point cloud processing module 1110: is used to acquire view images of the target scene from multiple perspectives, obtain sparse point cloud based on view images, and initialize an initial 3D representation model based on anchor points according to the sparse point cloud. The initial 3D representation model includes multiple anchor points and anchor point attributes corresponding to each anchor point. Anchor point attributes include feature attributes, size attributes and position offset.

[0167] Entropy coding module 1120: Used to compress and encode the anchor point attributes for each anchor point to obtain the encoded bitstream corresponding to the target scene.

[0168] Entropy decoding module 1130: used to perform entropy decoding on the encoded bitstream to obtain decoding feature attributes, decoding size attributes and decoding offset attributes.

[0169] Redundancy processing module 1140: used to estimate the mean of feature attributes using a preset entropy model, perform weighted prediction based on the decoded feature attributes and the mean of feature attributes to obtain weighted feature attributes, and group the weighted feature attributes by channel dimension to obtain rendering feature attributes.

[0170] Rendering and Reconstruction Module 1150: Used to generate a 3D representation model of the target scene based on rendering feature attributes, decoding size attributes, and decoding offset attributes.

[0171] The specific implementation of the three-dimensional scene reconstruction device in this embodiment is basically the same as the specific implementation of the three-dimensional scene reconstruction method described above, and will not be repeated here.

[0172] This application also provides an electronic device, including:

[0173] At least one memory;

[0174] At least one processor;

[0175] At least one program;

[0176] The program is stored in a memory, and the processor executes the at least one program to implement the three-dimensional scene reconstruction method described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0177] Please refer to Figure 12, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0178] The processor 1201 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0179] The memory 1202 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 using the three-dimensional scene reconstruction method of the embodiments of this application.

[0180] The input / output interface 1203 is used to implement information input and output;

[0181] The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and the bus 1205 is used to transmit information between the various components of the device (such as processor 1201, memory 1202, input / output interface 1203 and communication interface 1204).

[0182] The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0183] This application embodiment also provides a storage medium that stores a computer program, which, when executed by a processor, implements the above-described three-dimensional scene reconstruction method.

[0184] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0185] The 3D scene reconstruction method, apparatus, device, and storage medium proposed in this application acquire viewpoint images of the target scene from multiple perspectives, obtain sparse point clouds based on the viewpoint images, initialize an initial 3D representation model based on anchor points according to the sparse point clouds, the initial 3D representation model includes multiple anchor points and anchor point attributes corresponding to each anchor point, the anchor point attributes include feature attributes, size attributes, and position offsets, for each anchor point, the anchor point attributes are compressed and encoded to obtain an encoded bitstream corresponding to the target scene, then the encoded bitstream is entropy decoded to obtain decoded feature attributes, decoded size attributes, and decoded offset attributes, then weighted prediction is performed based on the decoded feature attributes and the mean of feature attributes predicted using a preset model to obtain weighted feature attributes, and the weighted feature attributes are grouped by channel dimension to obtain rendering feature attributes, finally generating a 3D representation model of the target scene based on the rendering feature attributes, decoded size attributes, and decoded offset attributes. In this application embodiment, the feature attributes of the anchor points are reasonably simplified. On the one hand, although the anchor point's feature attributes contain rich scene information, some of this information overlaps with the mean feature attributes predicted by the preset entropy model. Therefore, by introducing the mean feature attributes through a weighted prediction process, the anchor point's feature attributes only need to represent the missing or inaccurate scene information in the mean feature attributes, thus effectively reducing the scene information contained in the feature attributes. On the other hand, by analyzing the similarity between channels, it was found that the similarity between all channels is very high. Therefore, if divided into two groups, the similarity between the two groups will also be very high. Thus, residuals with low storage cost are used to model the subtle differences between the two groups. In other words, related technologies require storing two sets of features, while this application embodiment only requires storing one set of features and residuals. The other set of features can be represented by the stored set of features combined with the residuals. This reduces channel redundancy of feature attributes while retaining key channel information, further reducing the storage footprint of the anchor point's feature attributes. By optimizing the anchor point attributes, it is possible to reduce transmission bandwidth, reduce storage space, and improve the rate-distortion performance of the 3D representation model while obtaining higher quality reconstruction results.

[0186] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will understand that, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems. Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or a combination of certain steps, or different steps.

[0187] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0189] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0190] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0191] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the grouping of the units described above is only a logical functional grouping, and in actual implementation, there may be other grouping methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0192] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0194] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0195] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for reconstructing a three-dimensional scene, characterized in that, include: The target scene is obtained from multiple perspective images, and a sparse point cloud is obtained based on the perspective images. An initial 3D representation model based on anchor points is initialized based on the sparse point cloud. The initial 3D representation model includes multiple anchor points and anchor point attributes corresponding to each anchor point. The anchor point attributes include feature attributes, size attributes, and position offset. For each anchor point, the anchor point attributes are compressed and encoded using a preset entropy model to obtain the encoded bitstream corresponding to the target scene; Entropy decoding is performed on the encoded bitstream to obtain decoding feature attributes, decoding size attributes, and decoding offset attributes; We perform weighted prediction based on the decoded feature attributes and the mean of the feature attributes predicted based on the preset entropy model to obtain weighted feature attributes, and group the weighted feature attributes by channel dimension to obtain rendering feature attributes. A 3D representation model of the target scene is generated based on the rendering feature attributes, the decoding size attributes, and the decoding offset attributes.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The preset entropy model is a context model, including a binary hash table and a first multilayer perceptron. The step of using the preset entropy model to compress and encode the anchor attributes to obtain the encoded bitstream corresponding to the target scene includes: The context features are obtained by interpolating the coordinates of the anchor points in the binary hash table. The context features are input into the first multilayer perceptron for data prediction to obtain probability distribution parameters and quantization step size; The anchor point attribute is quantized using the quantization step size to obtain the quantized attribute, and the quantized attribute is entropy encoded using the probability distribution parameter to obtain the encoded bitstream.

3. The three-dimensional scene reconstruction method according to claim 2, characterized in that, The entropy decoding of the encoded bitstream to obtain decoding feature attributes, decoding size attributes, and decoding offset attributes includes: The encoded bitstream is entropy-decoded using the probability distribution parameters to obtain decoded quantization attributes, which include quantization feature attributes, quantization size attributes, and quantization position offset. Obtain the quantization step size corresponding to each of the decoded quantization attributes, multiply the quantization feature attribute and the quantization step size to obtain the decoded feature attribute, multiply the quantization size attribute and the quantization step size to obtain the decoded size attribute, and multiply the quantization position offset and the quantization step size to obtain the decoded offset attribute.

4. The three-dimensional scene reconstruction method according to claim 2, characterized in that, The step of performing weighted prediction based on the mean of the feature attributes and the decoded feature attributes to obtain weighted feature attributes includes: The context features are input into a second multilayer perceptron for data prediction to obtain feature channel fusion weights. Weighted feature attributes are obtained by performing weighted prediction based on the feature channel fusion weights, the mean of the feature attributes, and the decoded feature attributes.

5. The three-dimensional scene reconstruction method according to claim 4, characterized in that, The step of inputting the context features into a second multilayer perceptron for data prediction to obtain feature channel fusion weights includes: The contextual features are input into a second multilayer perceptron for data prediction to obtain a first intermediate value; The first intermediate value is input into a preset activation function for data calculation to obtain the second intermediate value; The difference between the first and the second intermediate value is obtained to obtain the feature channel fusion weight.

6. The three-dimensional scene reconstruction method according to claim 4, characterized in that, The weighted prediction based on the feature channel fusion weights, the mean of the feature attributes, and the decoded feature attributes to obtain weighted feature attributes includes: The product of the feature channel fusion weight and the decoded feature attribute is obtained to get the third intermediate value; Obtain the difference between the weights fused with the feature channels to get the intermediate weight value, and multiply the intermediate weight value by the mean of the feature attributes to get the fourth intermediate value; The weighted feature attribute is obtained by summing the third intermediate value and the fourth intermediate value.

7. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The step of grouping the weighted feature attributes by channel dimension to obtain rendering feature attributes includes: Based on a preset grouping method, the weighted feature attributes are divided into first channel features and second channel features in the channel dimension; The first channel feature and the second channel feature are added together to obtain the fifth intermediate value, and one of the first channel feature and the second channel feature is selected as the residual feature. The rendering feature attribute is obtained by splicing the non-residual feature and the fifth intermediate value.

8. A three-dimensional scene reconstruction device, characterized in that, include: Point cloud processing module: used to acquire view images of the target scene from multiple perspectives, and obtain sparse point cloud based on the view images. Initialize an anchor-based 3D representation model based on the sparse point cloud. The initial 3D representation model includes multiple anchor points and anchor point attributes corresponding to each anchor point. The anchor point attributes include feature attributes, size attributes, and position offset. Entropy coding module: used to compress and encode the attributes of each anchor point to obtain the encoded bitstream corresponding to the target scene; Entropy decoding module: used to perform entropy decoding on the encoded bitstream to obtain decoding feature attributes, decoding size attributes, and decoding offset attributes; Redundancy processing module: used to estimate the mean of feature attributes using a preset entropy model, perform weighted prediction based on the decoded feature attributes and the mean of feature attributes to obtain weighted feature attributes, and group the weighted feature attributes by channel dimension to obtain rendering feature attributes; Rendering and Reconstruction Module: Used to generate a 3D representation model of the target scene based on the rendering feature attributes, the decoding size attributes, and the decoding offset attributes.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the three-dimensional scene reconstruction method according to any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional scene reconstruction method according to any one of claims 1 to 7.

Images