Scene rendering method and apparatus

Abstract

The present application provides a scene rendering method and apparatus, which are used for performing three-dimensional scene modeling and rendering by taking a depth as prior, and have a more generalized three-dimensional scene rendering capability. The method comprises: acquiring a scene image, the scene image comprising data of a scene to be rendered; acquiring depth information of the scene image, wherein the depth information may be used for representing the depth corresponding to a pixel in the scene image, and may represent the distance between an object, in the image, corresponding to an actual scene and a camera; projecting the scene image into a space on the basis of the depth information to obtain point cloud data, wherein the depth information may be used as prior to determine the position of each object in the scene in the same space; extracting features from the point cloud data by means of a feature extraction network to obtain spatial features; inputting the spatial features into a Gaussian model to obtain three-dimensional Gaussian sphere data; and obtaining rendering data on the basis of the three-dimensional Gaussian sphere data.

Description

A scene rendering method and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202510203151.5, filed on February 20, 2025, entitled “A Scene Rendering Method and Apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer vision, and more particularly to a scene rendering method and apparatus. Background Technology

[0003] 3D scene rendering is a widely used solution, such as in autonomous driving, digital humans, or other visual applications. Traditional rendering methods involve explicitly simulating the models in the scene and then using a rendering engine to render the 3D scene models as images from a specific angle.

[0004] 3D Gaussian Splatting (3DGS) is an emerging scene reconstruction and image rendering scheme. Compared to neural volume rendering, it abandons the process of emitting a ray from each pixel and sampling a large number of rays. Instead, it represents the scene as a Gaussian sphere and obtains the rendering result by converting the data into a 3D Gaussian sphere, achieving real-time rendering speed. However, the 3DGS scheme is trained for each different scene before it can output the rendered scene image, thus its generalization ability is limited. Summary of the Invention

[0005] This application provides a scene rendering method and apparatus for using depth as a priori for 3D scene modeling and rendering, which has a more generalized 3D scene rendering capability.

[0006] In view of the above, in a first aspect, this application provides a scene rendering method, comprising: first, acquiring a scene image, the scene image including data of the scene to be rendered; then mapping the scene image into space according to depth information to obtain point cloud data, the depth information being used to represent the depth corresponding to pixels in the scene image, and to represent the distance between the object in the image corresponding to the actual scene and the camera, the depth information being used to determine the position of points in the point cloud data, that is, the depth information being used as a priori to determine the position of each object in the scene in the same space; then extracting features from the point cloud data through a feature extraction network to obtain spatial features; then inputting the spatial features into a Gaussian model to obtain three-dimensional Gaussian sphere data, the three-dimensional Gaussian sphere data including the values ​​of Gaussian parameters of at least one three-dimensional Gaussian sphere, the at least one three-dimensional Gaussian sphere being used to represent the scene to be rendered; and then obtaining rendering data based on the three-dimensional Gaussian sphere data.

[0007] In this embodiment, a 3D Gaussian sphere can be used to construct the scene during 3D scene reconstruction. During the construction of the 3D Gaussian sphere, the depth information corresponding to the image is used as a guide (i.e., prior) to determine the position of each point in the image within the same space, thus mapping the image into point cloud data. Therefore, when constructing the 3D Gaussian sphere subsequently, the geometric distribution relationship of each object in the scene within the same space can be determined based on this point cloud, extracting more accurate spatial features of the geometric distribution relationship. This allows the Gaussian model to construct a more accurate 3D Gaussian sphere based on these spatial features. This is equivalent to decoupling the modeling process, for example, by splitting it into a feature extraction network and a Gaussian model. By using depth information as a geometric prior, modeling can be achieved from a geometric perspective. Therefore, for different types of scenes, there is no need to use scene data separately for scene training. The spatial geometric distribution relationship of each object in the scene can be directly identified based on the input data using the method provided in this embodiment, thereby outputting rendering data with very strong generalization capabilities.

[0008] In one possible implementation, the three-dimensional Gaussian sphere data includes near-field Gaussian data, first color data, far-field Gaussian data, and second color data. The near-field Gaussian data includes the values ​​of the Gaussian parameters of the constructed near-field three-dimensional Gaussian sphere. The first color data is used to represent the color of the near-field three-dimensional Gaussian sphere, the second color data is used to represent the color of the far-field three-dimensional Gaussian sphere, and the far-field Gaussian data includes the values ​​of the Gaussian parameters of the far-field three-dimensional Gaussian sphere. The confidence level of the depth corresponding to the near-field three-dimensional Gaussian sphere is higher than the confidence level of the depth corresponding to the far-field three-dimensional Gaussian sphere. At least one three-dimensional Gaussian sphere includes both a far-field three-dimensional Gaussian sphere and a near-field three-dimensional Gaussian sphere.

[0009] Typically, the distribution characteristics of foreground and background in an image are different. For example, foreground may reflect light more strongly, so the corresponding depth is usually more accurate. Therefore, in this embodiment, foreground and background are modeled separately, so as to adaptively model for foreground and background separately to obtain foreground with richer details and background with more accurate color distribution.

[0010] In one possible implementation, the aforementioned process of inputting spatial features into a Gaussian model to obtain 3D Gaussian sphere data may include: inputting the spatial features into a decoder to output near-field Gaussian data, which includes the values ​​of the Gaussian parameters of the constructed near-field 3D Gaussian sphere; inputting the near-field Gaussian data into a near-field perceptron to output first color data, which represents the color of the near-field 3D Gaussian sphere; and collecting data from the aforementioned scene image based on the distance radius and inputting it into a distance perceptron to obtain distance Gaussian data and second color data, whereby the distance Gaussian data includes the values ​​of the parameters of the distance Gaussian sphere, and the second color data represents the color of the 3D Gaussian sphere. Therefore, different perceptrons are used for processing near and far scenes to achieve adaptive modeling of near and far scenes, resulting in Gaussian parameter values ​​that are more suitable for the scene.

[0011] In one possible implementation, the aforementioned near-view Gaussian data includes at least the Gaussian sphere position, scale value, and transparency value of the near-view 3D Gaussian sphere. The Gaussian sphere position is the location of the center point of the near-view 3D Gaussian sphere, the scale value represents the scale parameter value of the near-view 3D Gaussian sphere from different viewpoints, and the transparency value represents the transparency degree of the near-view 3D Gaussian sphere. The far-view Gaussian data includes at least the scale value of the far-view 3D Gaussian sphere, which represents the scale of the far-view 3D Gaussian sphere from different viewpoints. In this embodiment, the near-view portion can focus on the Gaussian sphere position, scale parameter, and transparency value, while the far-view portion mainly focuses on the scale parameter, thereby achieving more refined scene modeling for the near-view and better visual effect modeling for the far-view.

[0012] In one possible implementation, the aforementioned input of near-field Gaussian data into the near-field sensor and output of first color data can be achieved by: acquiring at least one first image frame from the scene image based on a first viewpoint, wherein the at least one first image frame is an image in the scene image adjacent to the first viewpoint; projecting a near-field three-dimensional Gaussian sphere onto the at least one first image frame according to camera parameters to obtain third color data, wherein the third color data includes the color of at least one sampling window under the first viewpoint; and then inputting the third color data into the near-field sensor and outputting the first color data.

[0013] In this embodiment of the application, in order to improve the color accuracy of the foreground part, the foreground Gaussian data can be projected onto the image collected in the real scene, thereby collecting the foreground color in the original data from various perspectives in the scene to obtain a more accurate foreground color.

[0014] In one possible implementation, the aforementioned method further includes: using depth information as a priori to obtain a visible map corresponding to a three-dimensional Gaussian sphere, wherein the visible map is the visible area of ​​the near-view three-dimensional Gaussian sphere from a first perspective, that is, the area visible within the scene from the first perspective; the aforementioned inputting the third color data into the near-view sensor and outputting the first color data may include: inputting the third color data and the visible map into the near-view sensor and outputting the first color data, thereby enabling the near-view sensor to output more accurate first color data by referring to the visible map.

[0015] In this embodiment, depth can be used as a priori to determine the visible area under the viewpoint, thereby modeling the scene for possible occlusion under each viewpoint and improving the accuracy of the constructed scenes.

[0016] In one possible implementation, the aforementioned process of using depth information as a priori to obtain a visible image may include: determining a relative vector based on the depth information, wherein the relative vector represents the distance and direction between the center point of the near-field three-dimensional Gaussian sphere and the current viewpoint position under the first viewpoint; determining a visible image based on the relative vector; and then inputting the third color data, the visible image, and the relative vector into the near-field sensor to output the first color data.

[0017] In this embodiment of the application, in case of possible occlusion at the viewing angle, a relative vector representing the direction of the viewing ray and the visible distance can be input into the near-field sensor to output more accurate color data in case of possible occlusion.

[0018] In one possible implementation, the aforementioned output of distant Gaussian data and second color data based on the distant radius and the distant sensor includes: acquiring at least one frame of a second image from a scene image based on a second viewpoint, wherein the at least one frame of the first image is an image in the scene image adjacent to the first viewpoint; projecting a distant three-dimensional Gaussian sphere onto the at least one frame of the second image according to camera parameters and the distant radius to obtain fourth color data, wherein the third color data includes the color of at least one sampling window under the first viewpoint; inputting the fourth color data into the distant sensor, and outputting the distant Gaussian data and the second color data.

[0019] In this embodiment, for the distant view, a three-dimensional Gaussian sphere of the distant view can be projected onto the original image based on the distant view radius, thereby obtaining the local color of the distant view in the real scene and inputting it into the distant view perceptron, thereby obtaining more accurate distant view Gaussian data and color data, and performing more targeted modeling of the distant view.

[0020] In one possible implementation, the aforementioned input of spatial features into the decoder to output near-field Gaussian data may include: sampling the spatial features multiple times to obtain multiple structured features; using the multiple structured features as input to the decoder, and iteratively processing to obtain near-field Gaussian data. In this embodiment, near-field Gaussian data can be output iteratively to obtain more accurate near-field Gaussian data.

[0021] In one possible implementation, the aforementioned method further includes: training an initial Gaussian model to obtain a Gaussian model, wherein the loss function used for training may include at least one of the following: the difference between the real color and the rendered color, the difference between the mean of the ground truth image and the mean of the rendered data, the standard deviation of the ground truth image and the rendered data, or the covariance of the ground truth image and the rendered data, wherein the rendered color includes first color data and second color data. In this embodiment, the model can be trained from dimensions such as color, image mean, or standard deviation, thereby improving the modeling effect output by the Gaussian model.

[0022] In one possible implementation, the aforementioned feature extraction network includes a network for extracting sparse features. Therefore, in this embodiment, spatial features can be extracted by extracting coefficient features, resulting in spatial features containing richer information, thereby improving the accuracy of the 3D scene obtained through subsequent scene modeling.

[0023] Secondly, this application provides a scene rendering apparatus, comprising:

[0024] The image acquisition module is used to acquire scene images, which include data of the scene to be rendered;

[0025] The mapping module is used to map the scene image into space based on the depth information of the scene image to obtain point cloud data. The depth information is used to determine the position of points in the point cloud data.

[0026] The feature extraction module is used to extract features from point cloud data through a feature extraction network to obtain spatial features;

[0027] The Gaussian modeling module is used to input spatial features into a Gaussian model to obtain three-dimensional Gaussian sphere data, which includes the values ​​of Gaussian parameters of at least one constructed three-dimensional Gaussian sphere.

[0028] The rendering module is used to obtain rendering data based on the 3D Gaussian sphere data.

[0029] The effects achieved by the second aspect and any optional implementation of the second aspect can be referred to the description of the first aspect or any optional implementation of the first aspect, and will not be repeated here.

[0030] In one possible implementation, the three-dimensional Gaussian sphere data includes near-view Gaussian data, first color data, far-view Gaussian data, and second color data. The near-view Gaussian data includes the values ​​of the Gaussian parameters of the constructed near-view three-dimensional Gaussian sphere. The first color data is used to represent the color of the near-view three-dimensional Gaussian sphere. The second color data is used to represent the color of the far-view three-dimensional Gaussian sphere. The far-view Gaussian data includes the values ​​of the Gaussian parameters of the far-view three-dimensional Gaussian sphere.

[0031] The confidence level of the depth corresponding to the near-view 3D Gaussian sphere is higher than that of the depth corresponding to the far-view 3D Gaussian sphere, and at least one 3D Gaussian sphere includes both the far-view 3D Gaussian sphere and the near-view 3D Gaussian sphere.

[0032] In one possible implementation, the aforementioned Gaussian modeling module is specifically used for: inputting spatial features into a decoder and outputting near-field Gaussian data; inputting the near-field Gaussian data into a near-field sensor and outputting first color data; and collecting data from the scene image based on the distant radius and inputting it into a distant sensor, outputting distant Gaussian data and second color data. The Gaussian model includes a decoder, a near-field sensor, and a distant sensor.

[0033] In one possible implementation, the aforementioned three-dimensional Gaussian sphere data includes near-view Gaussian data, first color data, far-view Gaussian data, and second color data, and the Gaussian model includes a decoder, a near-view sensor, and a far-view sensor.

[0034] In one possible implementation, the aforementioned near-field Gaussian data includes at least the Gaussian sphere position, scale value, and transparency value of the near-field three-dimensional Gaussian sphere; the far-field Gaussian data includes at least the scale value of the far-field three-dimensional Gaussian sphere.

[0035] In one possible implementation, the aforementioned Gaussian modeling module is specifically used to: acquire at least one first image frame corresponding to the first viewpoint, wherein the at least one first image frame is an image adjacent to the first viewpoint in the scene image; project a near-field three-dimensional Gaussian sphere onto the at least one first image frame according to camera parameters to obtain third color data, wherein the third color data includes the color of at least one sampling window under the first viewpoint; input the third color data into the near-field sensor and output the first color data.

[0036] In one possible implementation, the aforementioned Gaussian modeling module is further configured to: use depth information as a priori to obtain a visible map, which is the visible area of ​​a near-field three-dimensional Gaussian sphere from a first perspective; input the third color data and the visible map into a near-field perceptron, and output the first color data.

[0037] In one possible implementation, the aforementioned Gaussian modeling module is specifically used to: determine a relative vector based on depth information, wherein the relative vector represents the distance and direction between the center point of the near-field 3D Gaussian sphere and the current viewpoint position under the first viewpoint; determine a visible image based on the relative vector; input the third color data, the visible image, and the relative vector into the near-field sensor, and output the first color data.

[0038] In one possible implementation, the aforementioned Gaussian modeling module is specifically used to: acquire at least one frame of a second image corresponding to a first viewpoint, wherein the at least one frame of the second image is an image adjacent to the first viewpoint in the scene image; project a distant three-dimensional Gaussian sphere onto the at least one frame of the second image according to camera parameters to obtain fourth color data, wherein the third color data includes the color of at least one sampling window under the first viewpoint; input the fourth color data into the distant perception sensor and output the second color data.

[0039] In one possible implementation, the aforementioned Gaussian modeling module is specifically used to: sample the spatial features multiple times to obtain multiple structured features; and use the multiple structured features as input to the decoder to iteratively process and obtain near-field Gaussian data.

[0040] In one possible implementation, the aforementioned apparatus further includes a training module for training an initial Gaussian model to obtain a Gaussian model, wherein the loss function used for training includes at least one of the following: the difference between the real color and the rendered color, the difference between the mean of the ground truth image and the mean of the rendered data, the standard deviation of the ground truth image and the rendered data, or the covariance of the ground truth image and the rendered data; the rendered color includes first color data and second color data.

[0041] In one possible implementation, the aforementioned feature extraction network includes a network for extracting sparse features.

[0042] Thirdly, embodiments of this application provide a computing device including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the method shown in any of the first aspects above.

[0043] Fourthly, embodiments of this application provide a vehicle including at least one sensor, a processor, and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the method shown in any of the first aspects above.

[0044] Fifthly, embodiments of this application provide a digital processing chip or chip, the chip including a processing unit and a communication interface, the processing unit obtaining program instructions through the communication interface, the program instructions being executed by the processing unit, the processing unit being used to perform processing-related functions as described in the first aspect or any optional embodiment of the first aspect based on data collected by the at least one sensor.

[0045] In a sixth aspect, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any optional implementation thereof.

[0046] In a seventh aspect, embodiments of this application provide a computer program product comprising a computer program / instructions, which, when executed by a processor, causes the processor to perform the method described in the first aspect or any optional implementation thereof. Attached Figure Description

[0047] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0048] Figure 2 is a flowchart illustrating a scene rendering method provided in an embodiment of this application;

[0049] Figure 3 is a flowchart illustrating another scene rendering method provided in an embodiment of this application;

[0050] Figure 4 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0051] Figure 5 is a schematic diagram of another application scenario provided by an embodiment of this application;

[0052] Figure 6 is a schematic diagram of another application scenario provided by an embodiment of this application;

[0053] Figure 7 is a schematic diagram of the structure of a scene rendering device provided in an embodiment of this application;

[0054] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0056] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0057] To facilitate understanding, some concepts or categories involved in the embodiments of this application will be explained below.

[0058] (1) World Model

[0059] It refers to an internal representation or simulation structure used in fields such as artificial intelligence (AI), robotics, and computer vision to describe, understand, and predict the environment and its dynamic changes.

[0060] (2) Three-dimensional reconstruction

[0061] This refers to establishing a mathematical model of a three-dimensional object suitable for computer representation and processing. It forms the basis for processing, manipulating, and analyzing the properties of the object in a computer environment, essentially simulating a three-dimensional entity in a virtual three-dimensional space. Specifically, it involves determining the pose of each frame based on multiple images and the corresponding camera parameters, then mapping the pixel coordinates of each frame to three-dimensional space based on these poses, and finally reconstructing the three-dimensional model.

[0062] (3) Multilayer perceptron (MLP)

[0063] MLP (Multi-Level Processing) neural networks are a type of feedforward neural network (FFN). It is a fully connected (meaning each neuron is connected to all neurons in the previous layer) feedforward neural network model. For example, in classification problems, it transforms the raw scores for each class through multiple fully connected layers, and then uses a softmax function to obtain the predicted probability for each class. An MLP can be viewed as a directed graph composed of multiple node layers, each fully connected to the next. Besides the input node, each node is a neuron (or processing unit) with a non-linear activation function.

[0064] (4) transformer

[0065] A transformer architecture is a feature extraction network that includes both an encoder and a decoder. Of course, in some cases, a transformer architecture may not include an encoder but may include a decoder.

[0066] Encoder: Learns features, such as pixel features, in the global receptive field through self-attention.

[0067] Decoder: Learns the features of the desired module, such as the features of the output box, through self-attention and cross-attention.

[0068] For example, the structure of a Transformer layer in an existing scheme may include a multi-head attention network and a feedforward network module. Taking natural language processing as an example, the multi-head attention network obtains corresponding weight values ​​by calculating the relevance between words, thus obtaining context-related word representations, which is the core part of the Transformer structure. The feedforward network further transforms the obtained representations to obtain the final output of the Transformer layer. In addition to these two important components, residual layers (ADD) and linear normalization (Norm) are also stacked on these two components to optimize the output of the Transformer layer.

[0069] (5) SPARSE CNN

[0070] Sparse CNNs can be used to extract sparse features, specifically through network layers such as convolutional layers, pooling layers, and fully connected layers. The feature extraction process includes data preprocessing, feature extraction, and optional post-processing steps. During preprocessing, the input data can be standardized or normalized to ensure the data is on the same scale. Through these steps, Sparse CNNs can effectively extract useful features from the input data. In particular, sparse connections are implemented in the convolutional layers, meaning each output feature is connected to only a portion of the input feature map. This is key to Sparse CNNs reducing computational complexity and memory usage.

[0071] (6) Neural radiance field (NeRF)

[0072] NeRF is a method or model that uses neural networks to implicitly represent 3D scenes. A given scene can be learned using NeRF, and this scene is implicitly stored in the parameters of the NeRF neural network, i.e., the implicit representation of the scene. If a new perspective is needed, NeRF can be used to calculate the light and color values ​​at various locations within this scene, and after rendering, the new perspective can be output.

[0073] (7) 3D Gaussian Splatting (3DGS)

[0074] By using Gaussian functions to represent points or volumes in three-dimensional space, efficient and accurate representation of three-dimensional scenes can be achieved.

[0075] 3DGS uses Gaussian functions to represent points or volumes in three-dimensional space. Each point can be described by a Gaussian function, which defines the point's spatial location and its degree of diffusion along various axes. In 3DGS, the probability of a point's existence is determined by the value of the Gaussian function, allowing for the modeling of blurred boundaries on surfaces rather than strict geometric boundaries. 3DGS can represent three-dimensional shapes simultaneously at multiple scales, thus capturing details at different levels, from microscopic to macroscopic.

[0076] (8) Projection and Backprojection

[0077] Projection refers to the process of mapping geometric elements such as points, lines, and surfaces in three-dimensional space onto a two-dimensional plane. Projection typically refers to the process of mapping points in the world coordinate system (three-dimensional space) onto the image coordinate system (two-dimensional plane) using a camera model (such as a pinhole camera model or a fisheye camera model).

[0078] Backprojection is the inverse process of projection, that is, mapping a point on a two-dimensional plane back to three-dimensional space. Backprojection typically involves reconstructing the position of a point in three-dimensional space from the point in the image coordinate system using a camera model and corresponding algorithms.

[0079] The process of mapping scene images into space, as mentioned below in the embodiments of this application, is the back projection process.

[0080] The method provided in this application can be applied to various vision task scenarios, such as intelligent vehicle driving scenarios, drone flight scenarios, or intelligent vehicle driving scenarios. It can use input images to perform scene modeling and rendering, and output rendered data.

[0081] The method provided in this application can be applied to devices with display devices, such as vehicles or electronic devices, which can also be called computing devices. Specifically, the display device may include an electronic device display screen, a head-up display (HUD), a central control screen, an instrument panel screen, etc.

[0082] Traditional image generation methods based on computer graphics typically require extensive manual modeling. The quality of the generated images depends on the detail of the models in the scene and the capabilities of the image rendering engine. Furthermore, generating a large number of images requires significant investment in art and the accumulation of 3D assets, making the costs extremely high.

[0083] An existing image generation scheme based on Neural Radiance Fields (NeRF) uses data from real-world scene acquisitions for scene reconstruction, resulting in higher realism in scene reconstruction and image synthesis. This scheme optimizes the underlying continuous volumetric scene function by inputting a sparse set of views to synthesize new views of complex scenes. The scheme represents a scene using a fully connected (non-convolutional) deep network. Its input is a single continuous 5D coordinate (spatial position (x, y, z) and viewing direction (angle) (θ, φ)), and its output is the volumetric density at that spatial position and view-related color information. Views are synthesized by querying 5D coordinates along camera rays, and the output color and density are projected onto the image using classic volumetric rendering techniques. Since volumetric rendering is naturally differentiable, the only input required for optimization is a set of images with a known camera pose. However, the geometric consistency between image pixel ray sampling points is not well constrained. The network's depth output is obtained by integration using volumetric rendering, resulting in coarse depth predictions that often perform poorly in generating new viewpoint images and lack generalization ability. Furthermore, the direct input to the NeRF network is sample points on the light rays of the image after position encoding. The network optimization also involves training and estimating each sample point individually. In cases where the data perspective is sparse (such as in intelligent driving scenarios or large outdoor scenes), it is often difficult to recover good geometric information, resulting in poor rendering effects.

[0084] 3DGS is a novel scene reconstruction and image rendering scheme. Compared to volumetric rendering schemes, 3DGS abandons the process of emitting a ray from each pixel and sampling extensively on the rays. Instead, it represents the scene as a three-dimensional Gaussian sphere, obtaining rendering results by representing data as a three-dimensional Gaussian sphere, achieving real-time rendering speeds. By shifting from volume-based representation to learning point representation, it improves memory and runtime efficiency by more than an order of magnitude over existing schemes during training and inference. 3DGS starts with a uniformly sampled random point cloud, learns the spherical harmonic function coefficients related to the position and viewing direction of each point, and uses a differentiable splash-based renderer to project the point cloud onto the imaging plane according to a certain Gaussian distribution to obtain the rendering results. This is used to train the network to reproduce a set of input training images with a given camera pose. However, the 3DGS network requires separate training for each scene to achieve rendering capability, resulting in very weak generalization ability. Furthermore, it is prone to overfitting under sparse viewing perspectives, leading to poor rendering results.

[0085] Therefore, this application provides a scene rendering method and apparatus, which provides a scene rendering scheme with very strong generalization ability. It does not require separate training for each scene, and performs geometric traction modeling and rendering based on depth to obtain more accurate rendering data.

[0086] The system architecture and methodological steps provided in this application are described below.

[0087] First, the method provided in this application can be applied to 3D reconstruction scenarios. Specifically, it can be deployed in various electronic devices, such as server clusters, cloud platforms, personal computers, smartphones, or smart cars.

[0088] In one possible implementation, the method provided in this application can be deployed on computing devices such as personal computers, computer workstations, smartphones, tablets, laptops, and smart cars. Users can directly use computing devices to execute the method provided in the embodiments of this application to realize three-dimensional scene reconstruction.

[0089] For example, the method provided in this application can be applied to user devices, such as smartphones, tablets, or other devices with image processing capabilities. Users can use the user device to take photos, such as using the device's camera, depth camera, laser device, etc., to capture visual data, or directly extract images from storage space. Using the method provided in this application's embodiments, the visual data or image can be processed to output rendering data for a 3D scene. Furthermore, after obtaining the rendering data, it can be displayed on the user device's screen or a connected screen.

[0090] The method provided in this application can be deployed on server clusters, cloud platforms, or other devices with computing capabilities.

[0091] For example, the method provided in this application can be deployed in a cloud platform to provide services to users through cloud services.

[0092] For example, the cloud service system architecture used in the method provided in this application can be as shown in Figure 1. As shown in Figure 1, the cloud service system 10 may include a cloud 11 and a vehicle terminal 14. The number of vehicle terminals 14 may be one or more. Here, only one vehicle terminal is used as an example for illustration and is not intended to be limiting.

[0093] The cloud 11 may specifically include processing server 12 and data center 13.

[0094] The processing server 12 may specifically include a server cluster or a standalone computing device, or it may be other devices with computing capabilities. Optionally, the processing server 12 may cooperate with other computing devices, such as data storage, routers, load balancers, etc. The processing server 12 may use data in the data storage system or call program code in the data storage system to implement the method steps provided in the embodiments of this application.

[0095] Data center 13 can be used to store data for processing server 12 to query or write data, etc.

[0096] The vehicle-mounted device 14 establishes a connection with the processing server 12. The vehicle-mounted device 14 is equipped with one or more sensors, which can be used to collect perception data of the vehicle's environment. The number of vehicle-mounted devices 14 can be one or more. For example, in a scenario involving multiple vehicles, multiple vehicles can upload the collected data, such as images or point clouds, to the cloud server.

[0097] The vehicle terminal 14 can send the collected data and predicted vehicle driving decisions to the processing server 12. The processing server 12 can use the received data to perform 3D scene rendering using the method provided in this embodiment of the application, and then send the rendering data to the vehicle terminal. After receiving the rendering data, the vehicle terminal can display the rendering data on the display screen.

[0098] In summary, the method provided in this application embodiment can be applied to electronic devices, that is, the aforementioned cloud 11 can be various electronic devices, such as server clusters, cloud platforms, personal computers, smartphones or smart cars, etc.

[0099] Based on the architecture shown in Figure 1 above, the method provided in this application can be executed by a computing device, which may specifically include a server cluster or a cloud platform, and can provide services to users through a client, or the computing device may also be other devices with computing capabilities.

[0100] The method flow provided in the embodiments of this application will be described below in conjunction with the aforementioned architecture.

[0101] Referring to Figure 2, a flowchart of a scene rendering method provided in an embodiment of this application is shown below.

[0102] 201. Obtain scene image.

[0103] The scene image may include one or more frames. The scene image may be used to represent the scene to be rendered, or in other words, the scene image may contain data of the scene to be rendered.

[0104] Typically, when a scene image includes multiple frames, these frames can be images captured from different locations or using different camera poses. This provides more reference data for different rendering perspectives when rendering the scene.

[0105] The scene image can be data collected using an image sensor, data read from storage space, or received data, etc., depending on the actual application scenario.

[0106] For example, if the method provided in this application embodiment is deployed in an electronic device with an image sensor, the image sensor in the electronic device can be used directly to acquire images and obtain the scene image. For example, the method provided in this application embodiment can be deployed in a vehicle. During the vehicle's operation, the image sensor installed in the vehicle can be used to acquire environmental images, obtaining images acquired at different locations. The content contained in the images can then be modeled and rendered using the method provided in this application embodiment to obtain visual data representing the actual environment.

[0107] For example, if the method provided in this application embodiment is deployed in a cloud server, the scene image can be uploaded by an electronic device connected to the cloud server, or read by the cloud server from the data center, etc.

[0108] 202. Map the image into space based on the depth information of the scene image to obtain point cloud data.

[0109] The depth information can be used to represent the depth value of each point or region in the scene image. This depth value can also be understood as the distance between the object corresponding to a pixel in the image and the camera (i.e., the electronic device with the image sensor deployed) within the same space. Based on this depth, the position of each pixel in the image can be determined in the same space, and the data contained in the pixels in the image can be mapped to the same three-dimensional or multi-dimensional space to obtain point cloud data.

[0110] The specific method for obtaining depth information can be either by directly reading the depth image collected by the depth sensor or by performing depth estimation on the scene image. The specific method can be determined according to the actual application scenario, and this application does not limit it.

[0111] For example, in one possible implementation, when the image sensor is activated, the depth sensor can be activated simultaneously to collect data, and a corresponding depth map can be obtained at the same time as the scene image. This depth map can then correspond to the depth of each point in the scene image.

[0112] For example, in one possible implementation, a pre-trained depth estimation model can be used to estimate the depth of the input scene image to obtain at least one corresponding depth point cloud.

[0113] 203. Spatial features are obtained by extracting features from point cloud data through a feature extraction network.

[0114] Specifically, features can be extracted from point cloud data through a pre-trained feature extraction network to obtain spatial features, which are features that can characterize spatial relationships.

[0115] In one possible implementation, a sparse feature extraction network can be used to extract spatial features from point cloud data. This allows for more accurate feature extraction with lower computational cost. Compared to dense features, sparse features can contain richer content, thus providing a more accurate representation of the scene and improving the accuracy of subsequent scene modeling.

[0116] 204. Input the spatial features into the Gaussian model to obtain three-dimensional Gaussian sphere data.

[0117] Specifically, spatial features can be input into a Gaussian model to obtain three-dimensional Gaussian sphere data, which includes the values ​​of Gaussian parameters for at least one constructed three-dimensional Gaussian sphere. This Gaussian model can be used to construct three-dimensional Gaussian spheres based on the input spatial features; specifically, one or more three-dimensional Gaussian spheres can be constructed, where one of the spheres can represent a point or a region in space.

[0118] When only one 3D Gaussian sphere is constructed, this 3D Gaussian sphere can represent the entire scene; that is, the foreground 3D Gaussian sphere and the background 3D Gaussian sphere mentioned below can be the same 3D Gaussian sphere. When multiple 3D Gaussian spheres are constructed, different Gaussian spheres can represent different regions in the scene; that is, the foreground 3D Gaussian sphere and the background 3D Gaussian sphere mentioned below are different 3D Gaussian spheres.

[0119] In this embodiment of the application, to further adapt the rendering of foreground and background scenes more effectively, three-dimensional Gaussian spheres are constructed for both. That is, the aforementioned at least one three-dimensional Gaussian sphere can be divided into a foreground three-dimensional Gaussian sphere and a background three-dimensional Gaussian sphere. The foreground three-dimensional Gaussian sphere represents the foreground Gaussian sphere, and the background three-dimensional Gaussian sphere represents the background Gaussian sphere. The difference between foreground and background lies in the confidence level of their corresponding depths. For example, the confidence level of the depth of foreground is higher than that of the depth of background. Alternatively, it can be understood that foreground includes objects closer to the camera, and generally yields more accurate depth information, while background includes objects farther from the camera, and generally has lower depth accuracy than for foreground.

[0120] Optionally, the Gaussian model may specifically include one or more decoders, near-field perceptrons, or far-field perceptrons. The decoder can be used to decode the Gaussian parameters of the three-dimensional Gaussian sphere based on the input spatial features, obtaining the values ​​of the Gaussian parameters of the three-dimensional Gaussian sphere. The near-field perceptron can be used to process the features of the near-field and output the color data of the near-field, i.e., the first color data. The far-field perceptron can be used to process the input far-field data and output the color data of the far-field, i.e., the second color data.

[0121] It can be understood that the method provided in this application embodiment divides the scene into foreground and background for modeling and rendering respectively, and uses depth information as a geometric prior to accurately determine the coordinates of each Gaussian sphere in the foreground; for the background, more accurate color information is determined and modeled so that it can be combined to obtain more accurate rendering data including foreground and background.

[0122] In one possible implementation, spatial features can be input into a decoder to output near-field Gaussian data; subsequently, this near-field Gaussian data represents the Gaussian parameter values ​​of a constructed near-field 3D Gaussian sphere; the near-field Gaussian data is input into a near-field perceptron to output first color data, which represents the color of the near-field 3D Gaussian sphere; data collected from the aforementioned scene image based on the distance radius is input into a distance perceptron to obtain distance Gaussian data and second color data, where the distance Gaussian data represents the parameter values ​​of the distance Gaussian sphere and the second color data represents the color of the 3D Gaussian sphere; wherein the aforementioned 3D Gaussian sphere data includes near-field Gaussian data, first color data, distance Gaussian data, and second color data, and the aforementioned Gaussian model includes a decoder, a near-field perceptron, and a distance perceptron.

[0123] Therefore, in this embodiment, the foreground and background are modeled separately, so that different characteristics of the foreground and background can be modeled. For example, the depth information of the background may not be accurate, so the modeling is mainly focused on the outline or color of the object. The depth information of the foreground is usually more accurate, so the foreground can be modeled more accurately, so more accurate foreground and background data can be output.

[0124] Optionally, the spatial features can be sampled multiple times to obtain multiple structured features, such as through multiple interpolation and sampling. These multiple structured features are then used as input to the decoder, and iterative processing is performed to obtain near-field Gaussian data. In this embodiment, sampling structured features as input to the decoder can reduce noise that may be caused by inaccurate depth estimation, thereby iteratively updating to obtain more accurate near-field Gaussian data.

[0125] Optionally, the distant radius is the radius of the distant three-dimensional Gaussian sphere. This distant radius can be a preset value or a radius determined according to the confidence level of each depth in the depth information. For example, the depth with a confidence level lower than the preset confidence level can be used as the distant radius. Based on the distant radius, the distant Gaussian data and the second color data are output from the distant sensor.

[0126] Optionally, the 3D Gaussian sphere data may include, but is not limited to, the Gaussian sphere's position, scale value, transparency value, or spherical harmonic coefficients. The Gaussian sphere's position may include the coordinates of its center point; the scale parameter represents the 3D Gaussian sphere's scale from different viewpoints; the transparency value represents the 3D Gaussian sphere's transparency; and the spherical harmonic coefficients can represent the 3D Gaussian sphere's lighting characteristics. Typically, the 3D Gaussian sphere appears in different colors under different lighting environments or from different viewpoints. For ease of understanding, the spherical harmonic coefficients will be further broken down into color data for explanation.

[0127] Specifically, Gaussian data that is not entirely identical can be determined for the foreground and background. For example, in one possible implementation, the foreground Gaussian data includes at least the position of the Gaussian sphere, its scale value, and its transparency value; the background Gaussian data includes at least the scale value. Therefore, in this embodiment, different Gaussian data are output for the foreground and background, allowing for modeling at different dimensions. For the foreground, more detailed and accurate Gaussian data can be modeled, while for the background, since depth is usually inaccurate, modeling is mainly focused on dimensions such as scale or color. This avoids overfitting in the background and ultimately yields more accurate overall scene modeling data.

[0128] In one possible implementation, at least one first image frame is acquired from the scene image based on a first viewpoint. The first image frame is an image in the scene image adjacent to the first viewpoint. The first viewpoint can be any viewpoint in space, which can be understood as the viewpoint from which the scene to be rendered is viewed. A near-field three-dimensional Gaussian sphere is projected onto the at least one first image frame according to camera parameters to obtain third color data. The third color data includes the color of at least one sampling window under the first viewpoint. The third color data is input into the near-field sensor and the first color data is output.

[0129] The first-person perspective can be any perspective in the scene, or the current viewing perspective in a real-time rendered scene, and the number of first-person perspectives can be one or more. This application embodiment only describes the processing flow of one perspective as an example, and the specific process can be adjusted according to the actual application scenario.

[0130] The first-person perspective can be understood as the visible range of the image captured in the scene. When the scene image includes an image captured from the first-person perspective, the at least one first image frame can specifically be an image in the scene data corresponding to the first-person perspective. When the scene image does not include an image captured from the first-person perspective, the at least one first image frame can specifically be an image adjacent to the first-person perspective. Therefore, in the embodiments of this application, even for a perspective not used when capturing the image, three-dimensional Gaussian sphere modeling and rendering can be performed to achieve three-dimensional scene rendering under a new perspective.

[0131] In this embodiment, when outputting near-field color data, a near-field three-dimensional Gaussian sphere can be projected onto an adjacent image or an image of a visible area from the viewing angle. This allows for the acquisition of local color information (third color data) of windows in visible areas of different images from that viewing angle, based on the projection results. Color reconstruction is then performed based on this local color information to obtain the first color data. This achieves more accurate color reconstruction for the near-field area.

[0132] Optionally, depth information can be used as prior information to obtain a visible map corresponding to a 3D Gaussian sphere. This visible map represents the visible area of ​​at least one Gaussian sphere from a first-view perspective. Subsequently, the third color data and the visible map are input into a near-field sensor, which outputs the first color data. Therefore, in this embodiment, depth information can be used as prior information to identify the occlusion relationship between 3D Gaussian spheres and obtain a visible map from a first-view perspective. Thus, the occlusion relationship in the scene is considered during the modeling process, resulting in more accurate scene data that can represent the occlusion relationship between objects.

[0133] Optionally, a relative vector can be determined based on depth information. This relative vector represents the distance and direction between the position of the three-dimensional Gaussian sphere and the current viewpoint position from the first perspective. Then, a visible image is determined based on the relative vector. The third color data, the visible image, and the relative vector are then input into the near-field sensor, which outputs the first color data. In this embodiment, the distance and direction between the current viewpoint position and each three-dimensional Gaussian sphere can be determined based on depth information and used as input to the near-field sensor. This allows the near-field sensor to more accurately identify the relative positional and occlusion relationships between objects based on the input data, ultimately obtaining more accurate scene modeling data.

[0134] Optionally, for distant scene modeling, at least one second image from the scene image can be used based on a second viewpoint. This second image is an image adjacent to the first viewpoint within the scene image. Then, a 3D Gaussian sphere of the distant scene is projected onto the second image based on camera parameters and the distant radius to obtain fourth color data. This fourth color data includes the color of at least one sampling window from the first viewpoint, i.e., the color of the window in the distant scene. The fourth color data is then input into a distant scene perceptron, which outputs distant Gaussian data and the second color data. In other words, for the distant scene portion of the scene, the distant Gaussian data can also be projected onto adjacent images from the first viewpoint, thereby acquiring the color representation of the distant scene portion in the actual scene from that viewpoint and reconstructing more accurate distant scene color data, i.e., the second color data.

[0135] The second perspective can be the same as the aforementioned first perspective, or it can be a different perspective.

[0136] It should be noted that the aforementioned at least one first image and at least one second image can be the same image or different images. For example, when the first viewpoint and the second viewpoint are the same viewpoint, the at least one first image and at least one second image may be the same image.

[0137] 205. Rendering data is obtained based on the 3D Gaussian sphere data.

[0138] Once the 3D Gaussian sphere data is obtained, it can be rendered based on this data to output the corresponding rendered image. In other words, the 3D Gaussian sphere is projected onto an image based on the data to obtain a rendered image that represents the real scene.

[0139] In this embodiment, depth information can be used as a geometric prior for scene modeling. This allows the model to use scene-adaptive depth information as a geometric prior for 3D Gaussian sphere modeling, meaning that 3D Gaussian sphere modeling can be performed after acquiring depth information in different scenes, thus exhibiting very strong generalization ability.

[0140] Optionally, the initial Gaussian model can be trained to obtain a new Gaussian model. The loss function used during training can include at least one of the following: the difference between the real color and the rendered color, the difference between the mean of the ground truth image and the mean of the rendered data, the standard deviation of the ground truth image and the rendered data, or the covariance of the ground truth image and the rendered data. The rendered color can include the aforementioned first color data and second color data. Therefore, in this embodiment, the Gaussian model can utilize depth information as prior knowledge to model the scene from a geometrical perspective. The resulting model is highly generalizable, eliminating the need for separate model training for each scene.

[0141] The foregoing has described the method flow provided by the embodiments of this application. The method flow provided by the embodiments of this application will be further described below in conjunction with specific application scenarios.

[0142] First, referring to Figure 3, there is a schematic diagram of the architecture of the method provided in the embodiment of this application.

[0143] Specifically, the architecture can be divided into depth estimation, spatial feature extraction, near-field processing, and far-field processing, or near-field processing part and far-field processing part. The following sections will introduce depth estimation, near-field processing, and far-field processing respectively.

[0144] I. Depth Estimation

[0145] Depth estimation is an optional step. In this embodiment, depth estimation is used as an example to illustrate the process. In other scenarios, depth information can also be obtained through other means, such as through a depth sensor. This embodiment does not limit the scope of the application.

[0146] Depth estimation takes one or more frames of captured images as input; here, we'll use multiple frames as an example. Specifically, it estimates the distance of each pixel in one or more input images relative to the camera source. The specific depth estimation algorithm can be determined based on the camera used for the input images. For example, monocular depth estimation algorithms can be monocular or binocular, mapping the image to a depth map by fitting a function. Binocular estimation algorithms, for instance, can infer the distance between an object and the camera by calculating the pixel position difference (disparity) of the same point in the left and right images.

[0147] II. Close-up processing

[0148] The processing flow for the foreground can be divided into spatial feature extraction and foreground Gaussian data modeling.

[0149] First, we will introduce spatial feature extraction.

[0150] After obtaining depth information, multiple frames of images can be mapped to the same space based on this depth information to obtain point cloud data. Subsequently, a feature extraction network can be used to extract features from this point cloud data to obtain spatial features.

[0151] Specifically, SPARSE CNN can be used for feature extraction. SPARSE CNN extracts sparse features, resulting in spatial features that more accurately describe spatial characteristics, also known as global feature volume. Then, close-range Gaussian data modeling is performed based on these spatial features. The input to close-range Gaussian data modeling is the spatial features, and the output is the Gaussian data of the close-range portion.

[0152] Specifically, near-field Gaussian data modeling can employ one or more decoders and a near-field perceptron. The one or more decoders can be used to output partial Gaussian data of the near-field, such as at least one of the Gaussian sphere position, scale value, or transparency of the near-field 3D Gaussian sphere, while the near-field perceptron is used to output color data.

[0153] Optionally, when using the decoder to output the position, scale, or transparency values ​​of the near-field 3D Gaussian sphere, a more accurate set of these values ​​can be obtained through iteration. For example, the point cloud positions in the point cloud data can be used as the initial positions of the near-field 3D Gaussian sphere, as represented by: Typically, the point cloud data obtained by monocular depth estimation algorithms often contains noise. Therefore, during training, structured features are sampled multiple times in the global feature volume (GMV) using interpolation based on the position of the near-field 3D Gaussian sphere, serving as the perceptron MLP(D) data. opa D cov D pos The input of ) is processed through MLP(D opa D cov D pos Iteratively update the position μ of the near-field 3D Gaussian sphere. i , scale parameter ∑ i and transparency α i This is to ensure the accuracy of the geometry of the close-up 3D Gaussian sphere model.

[0154] Such as the MLP(D) opa D cov D pos This can be represented as:

[0155] Among them, D opa D cov With D pos These represent the decoders used to output transparency, scale parameters (also known as orientation scale), and position parameters, respectively. Δμ i μ represents the iteration process i The change value, μ init μ represents the output of the previous iteration. i This improves the accuracy of the final output three-dimensional Gaussian sphere data through iteration.

[0156] Then, based on depth information and MLP(D) opa D cov D pos The output of the close-up Gaussian data is further used to output color data.

[0157] Furthermore, for the foreground portion, to ensure more accurate high-frequency appearance information of the final modeled foreground 3D Gaussian sphere, a viewpoint can be selected as the target view. Based on the distance to this target viewpoint, N nearest reference view images (such as images of poses adjacent to the target viewpoint) are selected. Then, according to camera parameters, such as intrinsic or extrinsic parameters, the foreground 3D Gaussian sphere is projected onto these N nearest reference view images (i.e., the aforementioned first image), and a local color sample with a sampling window of W×W is acquired. The size of the sampling window can be preset or set according to the specific scenario.

[0158] Furthermore, regarding the foreground portion, considering the potential for occlusion in real-world scenes from a single viewpoint, as shown in Figure 4, the observed colors within the same foreground 3D Gaussian sphere may differ due to occlusion relationships between objects. Therefore, a depth map can be used as a priori to predict visibility maps for each foreground 3D Gaussian sphere.

[0159] Local color With visible graph As input to MLP(), the position and depth information of the near-view 3D Gaussian sphere are combined to determine the view ray in space at the current view position, which is then used as input to MLP() to further output color data adapted to the view.

[0160] For example, the specific calculation method for the output color parameter can be expressed as: δ ik =||μ i -t k ||2

[0161] v ik =(δ ik -d ik ) / δ ik

[0162] Where, μ i For the position of the near-field 3D Gaussian sphere, t k For pixel depth, δ i This represents the distance between the position of the Gaussian sphere in the near-field 3D and the pixel position. When predicting the spherical harmonic coefficient attributes of the near-field 3D Gaussian sphere, the relative direction of the near-field 3D Gaussian sphere with respect to the pixel center is used. As input to MLP(), this That is, the line connecting the current pixel to the center of the Gaussian sphere coordinate system.

[0163] This will output the final close-up portion in μ format. i ,∑ i α i c i .

[0164] In the geometry-guided near-field Gaussian data modeling provided in this application embodiment, the color point cloud converted from multiple frames of depth maps is used as the initial position of the near-field 3D Gaussian sphere. After extracting the Global Feature Volume through the spatially adaptive normalized SPARSE CNN designed within the near-field Gaussian data modeling, 3D features are obtained. The 3D features are used as the input of the near-field MLP to iteratively update the values ​​of parameters such as the position, scale, and transparency of the near-field 3D Gaussian sphere. Based on the target view, the near-field 3D Gaussian sphere is projected onto N frames of source view to extract image features of a certain window size, and combined with the predicted visibility map as the input of the MLP to predict the color data (spherical harmonic coefficients) of the Gaussian sphere.

[0165] In this embodiment, adaptive modeling is performed on the foreground portion, thereby extracting sparse features to obtain spatial features that better represent the spatial characteristics of the scene. This further enables more accurate 3D modeling of the scene and outputs a more structurally accurate foreground 3D scene.

[0166] III. Perspective Processing

[0167] For the distant view, this embodiment can set the radius corresponding to the distant view, use the radius as the Gaussian sphere radius of the distant view, and perform three-dimensional Gaussian sphere sampling on the sphere shell corresponding to the radius to model the distant view three-dimensional Gaussian sphere and obtain the three-dimensional scene of the distant view.

[0168] The radius can be a pre-defined fixed radius, or it can be a radius determined based on the depth values ​​or confidence levels of the depth values ​​at various points in the depth information. For example, the range where the depth value is greater than a first threshold or the confidence level of the depth value is not greater than a second threshold can be considered as the distant view. The specific radius can be determined according to the actual application scenario, and this application does not impose any restrictions on it.

[0169] The processing flow for distant views is similar to that for close-up views. The difference lies in the fact that 3D Gaussian sphere modeling for distant views emphasizes adjusting color c. bg With scale parameter s bg Typically, information about distant scenes may be inaccurate. Therefore, it is not necessary to focus on the position or transparency of the Gaussian sphere of the depth-related 3D Gaussian sphere. Instead, focus on color data and scale values ​​to construct a more visually accurate distant scene.

[0170] The use of a distance perception MLP (M bg ). Far-field sensor MLP (M bgThe input is a local color sampled in a W×W window, based on the Target View, projecting a distant 3D Gaussian sphere onto K frames of original view images (Source Views) (i.e., the aforementioned second image).

[0171] For example, it can be represented as:

[0172] Therefore, for the distant view, a 3D Gaussian sphere model of the distant view is performed using color data and scale values, thereby focusing on the color and scale of the distant view and obtaining a 3D distant view scene with better visual effects.

[0173] Compared to the aforementioned near-view 3D Gaussian sphere, the scale value of the near-view 3D Gaussian sphere may be the same as or different from that of the far-view 3D Gaussian sphere. The specific value can be calculated and determined based on the actual application scenario.

[0174] Furthermore, for the training portion of the Gaussian model, the loss function can be set as: L rgb =(1-λ) r L1+λ r L ssim

[0175] Among them, L rgb It can be set as the L2 loss function to measure the difference between the rendered color and the actual pixel color. ssim μ is the loss function used to measure the similarity between the real image and the rendered image. x μ y σ is the mean of the ground truth image and the rendered image. x σ y σ is the standard deviation between the ground truth image and the rendered image. xy C1 and C2 are constants, representing the covariance between the ground truth image and the rendered image.

[0176] When calculating the loss function, it's common practice to perform the calculation on a local window basis. For example, a local viewport can be opened, and the SSIM (Structural Similarity) value of the signal within that viewport can be calculated. The viewport is moved in increments of a certain number of pixels until the local structural similarity index for every location in the entire image has been calculated. The average of all local structural similarity losses then represents the structural similarity loss between the two images.

[0177] Therefore, in the embodiments of this application, the Gaussian model can be trained in a refined manner, and the loss function is calculated in units of local windows during the training process, so as to train the Gaussian model from the geometric dimension and obtain a Gaussian model with better generalization.

[0178] Furthermore, the method provided in this application can be specifically applied to the 3D reconstruction and image rendering process of 3D scenes (including static and dynamic scenes). For example, it can be applied to image generation frameworks that combine virtual and real elements, such as intelligent driving sensor simulation, virtual reality (VR), and augmented reality (AR). Image rendering is a core module in these applications, and the rendered image is a direct representation of the actual application scene. The realism of the rendered image can greatly affect the overall performance and user experience of the application. Using the method provided in this application, the foreground and background parts of the scene can be modeled separately, resulting in a more accurate 3D scene model.

[0179] Furthermore, in order to facilitate the demonstration of the effects achieved by the methods provided in the embodiments of this application, the effects achieved by the methods provided in the embodiments of this application will be introduced below using some specific application scenarios as examples.

[0180] Scene 1: Zero-shot scene reconstruction

[0181] Training is performed on N sequences on dataset A. Taking stereo images as an example, each sequence contains X sets of stereo images. 3D point clouds are obtained based on stereo depth estimation and input into Sparse CNN for training to obtain the Global Volume. Then, a generalizable scene reconstruction network is trained based on generalizable foreground and background Gaussian data modeling.

[0182] Inference is performed on any new sequence in dataset B, where each sequence contains Y sets of monocular images, as shown in Figure 5. Any image (Y>=1) in this sequence is input, and a 3D point cloud is obtained based on monocular depth estimation. This point cloud is then input into Sparse CNN for inference to obtain the Global Volume. Subsequently, based on generalizable foreground and background Gaussian data modeling, new scene reconstruction and rendering results are directly obtained. Therefore, it is not necessary to train separately for each scene, and more generalized 3D scene reconstruction can be achieved.

[0183] Therefore, for the zero-shot scene reconstruction problem, for the first time, generalizable (for the reconstruction of new scenes, no retraining is required, only inference) GS (gaussian splatting) scene reconstruction and rendering has been realized.

[0184] Scenario 2: Scene Reconstruction with Few Shots

[0185] The network is trained on dataset A with N sequences, each containing X sets of stereo images. 3D point clouds are obtained based on stereo depth estimation and input into Sparse CNN to obtain the Global Volume. Then, a generalizable scene reconstruction network is trained based on generalizable foreground and background Gaussian data modeling.

[0186] Infer on any new sequence in the B dataset (Waymo). Each sequence contains Y sets of monocular images. Input any image (y>=1) in this sequence, obtain a 3D point cloud based on monocular depth estimation, and input it into 3D SPADE CNN inference to obtain the Global Volume.

[0187] The decoder MLP fine-tunes (<1000 steps) on the sparse input image. Generalizable foreground and background Gaussian data modeling is then used for forward inference to obtain new scene reconstruction and rendering results.

[0188] For example, Figure 6 shows a comparison of the rendering effects of the method provided in this application embodiment with existing solutions. Clearly, the peak signal-to-noise ratio (PNSR) and learned perceptual image patch similarity (LPIPS) achieved by the method provided in this application embodiment are superior to those achieved by existing NeRF (i.e., Nerfacto shown in Figure 6) and 3DGS solutions. The method provided in this application embodiment can achieve more accurate 3D scene modeling.

[0189] Therefore, for the problem of reconstructing sparse view scenes with few shots, the reconstruction efficiency and reconstruction and rendering accuracy surpass the sparse view state-of-the-art (SOTA) solution.

[0190] Therefore, this application provides a generalizable 3D scene modeling scheme for outdoor unbounded scenes. It structurally decouples the scene into foreground and background, and designs corresponding generalizable Gaussian models with geometry guidance for different regions. Furthermore, it provides a color point cloud obtained by accumulating depth maps, extracts the Global Feature Volume using Sparse CNN for foreground Gaussian data modeling, and uses the color point cloud as the initial geometry of the foreground Gaussian model, optimizing its geometric properties iteratively. For monocular estimation scenes, the depth map is used as a priori estimate of occlusion relationships to estimate the foreground appearance, significantly improving the geometric reconstruction and rendering quality of the foreground. For the background, a generalizable background reconstruction model can be obtained based on fixed-radius spherical shell sampling, showing significant improvements in depth and 3D geometry compared to existing schemes.

[0191] The foregoing has described the method flow provided in the embodiments of this application. The structure of the apparatus for executing the foregoing method flow provided in the embodiments of this application is described below.

[0192] Referring to Figure 7, a schematic diagram of the structure of the mid-scene rendering device provided in this application includes:

[0193] The image acquisition module 701 is used to acquire a scene image, which includes data of the scene to be rendered.

[0194] The mapping module 702 is used to map the scene image into space based on the depth information to obtain point cloud data. The depth information is used to determine the position of points in the point cloud data.

[0195] The feature extraction module 703 is used to extract features from point cloud data through a feature extraction network to obtain spatial features;

[0196] Gaussian modeling module 704 is used to input spatial features into a Gaussian model to obtain three-dimensional Gaussian sphere data, which includes the values ​​of Gaussian parameters of at least one constructed three-dimensional Gaussian sphere.

[0197] Rendering module 705 is used to obtain rendering data based on three-dimensional Gaussian sphere data.

[0198] In one possible implementation, the three-dimensional Gaussian sphere data includes near-view Gaussian data, first color data, far-view Gaussian data, and second color data. The near-view Gaussian data includes the values ​​of the Gaussian parameters of the constructed near-view three-dimensional Gaussian sphere. The first color data is used to represent the color of the near-view three-dimensional Gaussian sphere. The second color data is used to represent the color of the far-view three-dimensional Gaussian sphere. The far-view Gaussian data includes the values ​​of the Gaussian parameters of the far-view three-dimensional Gaussian sphere.

[0199] The confidence level of the depth corresponding to the near-view 3D Gaussian sphere is higher than that of the depth corresponding to the far-view 3D Gaussian sphere, and at least one 3D Gaussian sphere includes both the far-view 3D Gaussian sphere and the near-view 3D Gaussian sphere.

[0200] In one possible implementation, the aforementioned Gaussian modeling module 704 is specifically used for: inputting spatial features into the decoder and outputting near-field Gaussian data; inputting the near-field Gaussian data into the near-field sensor and outputting first color data; collecting data from the scene image according to the far-field radius and inputting it into the far-field sensor, outputting far-field Gaussian data and second color data. The Gaussian model includes a decoder, a near-field sensor, and a far-field sensor.

[0201] In one possible implementation, the aforementioned near-field Gaussian data includes at least the Gaussian sphere position, scale value, and transparency value of the near-field three-dimensional Gaussian sphere; the far-field Gaussian data includes at least the scale value.

[0202] In one possible implementation, the aforementioned Gaussian modeling module 704 is specifically used to: acquire at least one first image frame corresponding to the first viewpoint, wherein the at least one first image frame is an image adjacent to the first viewpoint in the scene image; project a near-field three-dimensional Gaussian sphere onto the at least one first image frame according to camera parameters to obtain third color data, wherein the third color data includes the color of at least one sampling window under the first viewpoint; input the third color data into the near-field sensor and output the first color data.

[0203] In one possible implementation, the aforementioned Gaussian modeling module 704 is further configured to: use depth information as a priori to obtain a visible map, which is the visible area of ​​a near-field three-dimensional Gaussian sphere from a first perspective; input the third color data and the visible map into the near-field perceptron, and output the first color data.

[0204] In one possible implementation, the aforementioned Gaussian modeling module 704 is specifically used to: determine a relative vector based on depth information, wherein the relative vector represents the distance and direction between the center point of the three-dimensional Gaussian sphere and the current viewpoint position under the first viewpoint; determine a visible image based on the relative vector; input the third color data, the visible image and the relative vector into the near-field sensor, and output the first color data.

[0205] In one possible implementation, the aforementioned Gaussian modeling module 704 is specifically used to: acquire at least one frame of a second image corresponding to a first viewpoint, wherein the at least one frame of the second image is an image adjacent to the first viewpoint in the scene image; project a distant three-dimensional Gaussian sphere onto the at least one frame of the second image according to camera parameters to obtain fourth color data, wherein the third color data includes the color of at least one sampling window under the first viewpoint; input the fourth color data into the distant perception sensor and output the second color data.

[0206] In one possible implementation, the aforementioned Gaussian modeling module 704 is specifically used to: sample the spatial features multiple times to obtain multiple structured features; use the multiple structured features as input to the decoder and iteratively process them to obtain near-field Gaussian data.

[0207] In one possible implementation, the aforementioned apparatus further includes a training module 706 for training an initial Gaussian model to obtain a Gaussian model, wherein the loss function used for training includes at least one of the following: the difference between the real color and the rendered color, the difference between the mean of the ground truth image and the mean of the rendered data, the standard deviation of the ground truth image and the rendered data, or the covariance of the ground truth image and the rendered data; the rendered color includes first color data and second color data.

[0208] In one possible implementation, the aforementioned feature extraction network includes a network for extracting sparse features.

[0209] Figure 8 shows a schematic diagram of the hardware structure of a computing device 80 provided in an embodiment of this application. This computing device 80 can be used to implement the steps of the methods shown in Figures 2 to 6, and may specifically include the aforementioned electronic device.

[0210] The computing device 80 shown in Figure 8 may include a processor 801, a memory 802, a communication interface 803, and a bus 804. The processor 801, the memory 802, and the communication interface 803 can be connected via the bus 804.

[0211] The processor 801 is the control center of the computing device 80. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adapted to the actual application scenario.

[0212] As an example, processor 801 may include one or more CPUs, and may also include other processors, such as the CPU, NPU or GPU shown in Figure 8.

[0213] The memory 802 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0214] In one possible implementation, the memory 802 can exist independently of the processor 801. The memory 802 can be connected to the processor 801 via a bus 804 and is used to store data, instructions, or program code. When the processor 801 calls and executes the instructions or program code stored in the memory 802, it can implement the methods provided in the embodiments of this application, such as the methods shown in Figures 2 to 6.

[0215] In another possible implementation, the memory 802 can also be integrated with the processor 801.

[0216] The communication interface 803 is used for the computing device 80 to connect with other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 803 may include a receiving unit for receiving data and a transmitting unit for sending data.

[0217] Bus 804 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 8, but this does not indicate that there is only one bus or one type of bus.

[0218] It should be noted that the structure shown in FIG8 does not constitute a limitation on the computing device 80. In addition to the components shown in FIG8, the computing device 80 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0219] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a data quantization device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0220] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0221] This application also provides a computer-readable storage medium storing a program for training a model or performing inference tasks, which, when run on a computer, causes the computer to perform all or part of the steps in the methods described in the embodiments shown in Figures 2 to 6 above.

[0222] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the method steps of any one or more of the foregoing embodiments based on the program code stored in the external memory.

[0223] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0224] The apparatus provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip to execute the methods described in the embodiments shown in Figures 2 to 6. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can also be a storage unit located outside the chip within an electronic device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0225] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0226] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0227] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0228] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0229] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0230] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved. The division of modules in this application is a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs.

Claims

1. A method of scene rendering, characterized by, include: Acquire a scene image, which includes data of the scene to be rendered; The scene image is mapped into space based on the depth information of the scene image to obtain point cloud data. The depth information is used to determine the position of points in the point cloud data. Spatial features are obtained by extracting features from the point cloud data using a feature extraction network; The spatial features are input into a Gaussian model to obtain three-dimensional Gaussian sphere data. The three-dimensional Gaussian sphere data includes the values ​​of Gaussian parameters of at least one three-dimensional Gaussian sphere, which is used to represent the scene to be rendered. Rendering data is obtained based on the three-dimensional Gaussian sphere data.

2. The method according to claim 1, characterized in that, The three-dimensional Gaussian sphere data includes near-view Gaussian data, first color data, far-view Gaussian data, and second color data. The near-view Gaussian data includes the value of the Gaussian parameter of the near-view three-dimensional Gaussian sphere. The first color data is used to represent the color of the near-view three-dimensional Gaussian sphere. The second color data is used to represent the color of the far-view three-dimensional Gaussian sphere. The far-view Gaussian data includes the value of the Gaussian parameter of the far-view three-dimensional Gaussian sphere. The confidence level of the depth corresponding to the near-view 3D Gaussian sphere is higher than that of the depth corresponding to the far-view 3D Gaussian sphere, and the at least one 3D Gaussian sphere includes the far-view 3D Gaussian sphere and the near-view 3D Gaussian sphere.

3. The method of claim 2, wherein, The step of inputting the spatial features into a Gaussian model to obtain three-dimensional Gaussian sphere data includes: The spatial features are input into the decoder, and the near-field Gaussian data is output. The near-field Gaussian data is input into the near-field sensor, and the first color data is output. Data is collected from the scene image based on the distant radius and input to the distant perception device, and the distant Gaussian data and the second color data are output; the Gaussian model includes the decoder, the near perception device and the distant perception device.

4. The method according to claim 3, characterized in that, The near-field Gaussian data includes at least the position, scale value, and transparency value of the near-field three-dimensional Gaussian sphere; The prospective Gaussian data includes at least the scale value of the prospective three-dimensional Gaussian sphere.

5. The method according to claim 3 or 4, characterized in that, The step of inputting the near-field Gaussian data into the near-field sensor and outputting the first color data includes: At least one first image frame is acquired based on a first perspective, wherein the at least one first image frame is an image in the scene image that is adjacent to the first perspective; The near-field 3D Gaussian sphere is projected onto the at least one frame of the first image according to the camera parameters to obtain third color data, the third color data including the color of at least one sampling window under the first viewpoint; The third color data is input into the near-field sensor, and the first color data is output.

6. The method of claim 5, wherein, The method further includes: Using the depth information as a priori, a visible map is obtained, which is the visible area of ​​the near-field three-dimensional Gaussian sphere from the first viewpoint; The step of inputting the third color data into the near-field sensor and outputting the first color data includes: The third color data and the visible image are input into the near-field sensor, and the first color data is output.

7. The method of claim 6, wherein, The step of using the depth information as a priori to obtain the visible map includes: A relative vector is determined based on the depth information, and the relative vector represents the distance and direction between the center point of the near-field three-dimensional Gaussian sphere and the current viewpoint position under the first viewpoint. The visible graph is determined based on the relative vector; The step of inputting the third color data and the visible image into the near-field sensor and outputting the first color data includes: The third color data, the visible image, and the relative vector are input into the near-field sensor, and the first color data is output.

8. The method according to any one of claims 3-7, characterized in that, The step of outputting distant Gaussian data and the second color data based on the distant radius and the distant sensor includes: At least one frame of a second image is acquired based on a second perspective, wherein the at least one frame of the second image is an image in the scene image that is adjacent to the second perspective; The distant three-dimensional Gaussian sphere is back-projected onto the at least one frame of the second image based on the camera parameters and the distant radius to obtain the fourth color data. The third color data includes the color of at least one sampling window under the first viewpoint. The fourth color data is input into the distant sensor, and the distant Gaussian data and the second color data are output.

9. The method according to any one of claims 3-8, characterized in that, The step of inputting the spatial features into the decoder and outputting near-field Gaussian data includes: Multiple structural features are obtained by sampling the spatial features multiple times; The multiple structured features are used as input to the decoder, and the near-field Gaussian data is obtained through iterative processing.

10. The method according to any one of claims 2-9, characterized in that, The method further includes: The initial Gaussian model is trained to obtain the Gaussian model, wherein the loss function used in the training includes at least one of the following: the difference between the real color and the rendered color, the difference between the mean of the ground truth image and the mean of the rendered data, the standard deviation of the ground truth image and the rendered data, or the covariance of the ground truth image and the rendered data; the rendered color includes first color data and second color data.

11. The method according to any one of claims 1-10, characterized in that, The feature extraction network includes a network for extracting sparse features.

12. A scene rendering apparatus, characterized by comprising: include: An image acquisition module is used to acquire a scene image, wherein the scene image includes data of the scene to be rendered; The projection module is used to map the scene image into space based on the depth information of the scene image to obtain point cloud data, wherein the depth information is used to determine the position of points in the point cloud data; The feature extraction module is used to extract features from the point cloud data through a feature extraction network to obtain spatial features; A Gaussian modeling module is used to input the spatial features into a Gaussian model to obtain three-dimensional Gaussian sphere data. The three-dimensional Gaussian sphere data includes the values ​​of Gaussian parameters of at least one three-dimensional Gaussian sphere, and the at least one three-dimensional Gaussian sphere is used to represent the scene to be rendered. The rendering module is used to obtain rendering data based on the three-dimensional Gaussian sphere data.

13. The apparatus according to claim 12, characterized in that, The three-dimensional Gaussian sphere data includes near-view Gaussian data, first color data, far-view Gaussian data, and second color data. The near-view Gaussian data includes the value of the Gaussian parameter of the near-view three-dimensional Gaussian sphere. The first color data is used to represent the color of the near-view three-dimensional Gaussian sphere. The second color data is used to represent the color of the far-view three-dimensional Gaussian sphere. The far-view Gaussian data includes the value of the Gaussian parameter of the far-view three-dimensional Gaussian sphere. The confidence level of the depth corresponding to the near-view 3D Gaussian sphere is higher than that of the depth corresponding to the far-view 3D Gaussian sphere, and the at least one 3D Gaussian sphere includes the far-view 3D Gaussian sphere and the near-view 3D Gaussian sphere.

14. The apparatus of claim 13, wherein, The Gaussian modeling module is specifically used for: The spatial features are input into the decoder, and the near-field Gaussian data is output. The near-field Gaussian data is input into the near-field sensor, and the first color data is output. Data is collected from the scene image based on the distant radius and input to the distant perception device, and the distant Gaussian data and the second color data are output; the Gaussian model includes the decoder, the near perception device and the distant perception device.

15. The apparatus according to claim 14, characterized in that, The near-field Gaussian data includes at least the position, scale value, and transparency value of the near-field three-dimensional Gaussian sphere; The prospective Gaussian data includes at least the scale value of the prospective three-dimensional Gaussian sphere.

16. The apparatus of claim 13 or 14, wherein, The Gaussian modeling module is specifically used for: At least one first image frame is acquired based on a first perspective, wherein the at least one first image frame is an image in the scene image that is adjacent to the first perspective; The near-field 3D Gaussian sphere is projected onto the at least one frame of the first image according to the camera parameters to obtain third color data, the third color data including the color of at least one sampling window under the first viewpoint; The third color data is input into the near-field sensor, and the first color data is output.

17. The apparatus of claim 16, wherein, The Gaussian modeling module is also used for: Using the depth information as a priori, a visible map is obtained, which is the visible area of ​​the near-field three-dimensional Gaussian sphere from the first viewpoint; The third color data and the visible image are input into the near-field sensor, and the first color data is output.

18. The apparatus of claim 17, wherein, The Gaussian modeling module is specifically used for: A relative vector is determined based on the depth information, and the relative vector represents the distance and direction between the center point of the near-field three-dimensional Gaussian sphere and the current viewpoint position under the first viewpoint. The visible graph is determined based on the relative vector; The third color data, the visible image, and the relative vector are input into the near-field sensor, and the first color data is output.

19. The apparatus of any one of claims 14-18, wherein, The Gaussian modeling module is specifically used for: At least one frame of a second image is acquired based on a second perspective, wherein the at least one frame of the second image is an image in the scene image that is adjacent to the first perspective; The distant three-dimensional Gaussian sphere is back-projected onto the at least one frame of the second image based on the camera parameters and the distant radius to obtain fourth color data, the fourth color data including the color of at least one sampling window under the first viewpoint; The fourth color data is input into the distant sensor, and the distant Gaussian data and the second color data are output.

20. The apparatus of any one of claims 13-18, wherein, The Gaussian modeling module is specifically used for: Multiple structural features are obtained by sampling the spatial features multiple times; The multiple sampled features are used as input to the decoder, and the near-field Gaussian data is obtained through iterative processing.

21. The apparatus of any of claims 13-20, wherein, The device further includes: The training module is used to train the initial Gaussian model to obtain the Gaussian model. The loss function used in the training includes at least one of the following: the difference between the real color and the rendered color, the difference between the mean of the ground truth image and the mean of the rendered data, the standard deviation of the ground truth image and the rendered data, or the covariance of the ground truth image and the rendered data; the rendered color includes first color data and second color data.

22. The apparatus of any one of claims 12-21, wherein, The feature extraction network includes a network for extracting sparse features.

23. A computing device, comprising: The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the computing device performs the method as described in any one of claims 1-11.

24. A vehicle characterized by comprising: The device includes at least one sensor, a memory, and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the device performs the steps of the method as described in any one of claims 1-11 based on data acquired by the at least one sensor.

25. A computer storage medium, comprising, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 11.

26. A computer program product, characterised in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 11.

Images