Monocular visual slam method based on 3D gaussian splatting and neural network

Abstract

The present application provides a monocular visual SLAM method based on 3D Gaussian splatting and a neural network. Monocular depth estimation is performed on selected keyframes by using a neural network, and depth results are used for optimization of 3D Gaussian primitives; in addition, the present application introduces a global saliency score to measure the importance of primitives in a 3D Gaussian scene; the global saliency score and opacity are used as comprehensive criteria for evaluating the importance of 3D Gaussian primitives, and, by means of the comprehensive evaluation, rendering details are preserved, thereby achieving higher-quality rendering results.

Description

A 3D Gaussian sputtering monocular vision SLAM method incorporating neural networks Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a 3D Gaussian sputtering monocular vision SLAM method that combines neural networks. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a crucial research area in computer vision and robotics, enabling robots to gradually build a map of their surroundings while determining their own position and orientation in unknown environments. SLAM was initially applied to robotics and has since been extended to augmented reality, autonomous driving, and other fields. Visual SLAM, a subclass of SLAM, primarily calculates the camera's motion trajectory corresponding to the current video by processing consecutive RGB-D or RGB video frames, and combines the camera pose with the corresponding video frame to construct a map. Visual SLAM utilizes information acquired from monocular, binocular, or RGB-D cameras to locate the camera itself and reconstruct the surrounding environment, offering advantages such as lower cost, lightweight hardware, and diverse application scenarios.

[0003] 3D Gaussian Splatting is an explicit radiative field technique for building high-quality 3D scenes that can be rendered quickly. A 3D Gaussian scene is composed of 3D Gaussian primitives, each containing a center point position, a covariance matrix controlling the primitive's shape, opacity, and spherical harmonic coefficients for color calculation. By receiving RGB images from multiple viewpoints and their corresponding camera poses as input, a loss function is calculated between the rendered 3D Gaussian scene and the ground truth RGB image, continuously adjusting the properties of each 3D Gaussian primitive in the scene. After training, the 3D Gaussian scene can produce high-quality renderings of new views.

[0004] Matsuki, H. et al. proposed a visual SLAM system combining 3D Gaussian sputtering. In monocular mode, when tracking camera pose, the system selects keyframes from the received RGB image by calculating co-view and maintains an optimization window to ensure system speed. During tracking, a certain amount of Gaussian noise is added to a fixed depth to initialize the depth, which also serves as the center point of the initial primitives in the 3D Gaussian scene. During mapping, based on the established scene, the system performs 3D Gaussian primitive densification and opacity-based pruning to optimize the properties of each 3D Gaussian primitive in the scene; at the same time, to suppress the absolute scale runaway of 3D Gaussian primitives, the system introduces a regularization method for suppression. By calculating the loss function between the rendered result of the established 3D Gaussian scene and the input RGB image, the system continuously optimizes the camera pose and the 3D Gaussian scene, ultimately obtaining the camera movement trajectory and the 3D Gaussian scene result that can generate new view rendering results. Compared to traditional visual SLAM, which typically only generates geometric scene mapping, the introduction of the 3D Gaussian sputtering method makes the scene rendering results of this system anisotropic with the camera pose, thereby obtaining high-quality new view rendering results.

[0005] Fan, Z, and others proposed a novel 3D Gaussian primitive pruning strategy, establishing a global saliency score to comprehensively assess the importance of 3D Gaussian primitives. By calculating and ranking the global saliency scores of all 3D Gaussian primitives, and then calculating a threshold according to a predetermined ratio, all 3D Gaussian primitives with global saliency scores below the threshold are pruned. Compared to pruning solely based on opacity, which results in the loss of detail, the global saliency score more comprehensively considers factors affecting 3D Gaussian primitive rendering, preserving 3D Gaussian primitives with low opacity but significant rendering importance, making it a more reasonable pruning strategy.

[0006] Visual SLAM has wide applications in smart homes, augmented reality, and robotics. Matsuki et al. were among the first to propose fusing 3D Gaussian sputtering with visual SLAM, enabling the visual SLAM system to produce high-quality new view rendering results. However, this system still has certain drawbacks. The lack of depth information in monocular visual SLAM affects the accuracy of camera pose tracking and also impacts the center point position during 3D Gaussian scene initialization, leading to a decrease in mapping performance. Furthermore, during the mapping optimization process, the system still only uses the strategies from the original 3D Gaussian sputtering method, using opacity as the sole 3D Gaussian primitive pruning criterion, which results in a significant loss of scene detail in the rendered result. Summary of the Invention

[0007] To address the aforementioned issues, this application provides a 3D Gaussian sputtering monocular vision SLAM method that incorporates neural networks. It introduces a global saliency score to measure the importance of primitives in a 3D Gaussian scene, enabling the pruning method to retain more important primitives and obtain higher quality rendering results.

[0008] A 3D Gaussian sputtering monocular vision SLAM method incorporating neural networks includes:

[0009] S1: Obtain the co-visibility between the current RGB video frame and the latest keyframe in the keyframe window;

[0010] S2: Determine if the co-visibility is below the threshold. If yes, include the current RGB video frame in the keyframe window as the latest keyframe, and perform monocular depth estimation on the latest keyframe to obtain the predicted depth value corresponding to each pixel position on the latest keyframe before proceeding to S3; if no, proceed to S4.

[0011] S3: Use the pixel positions on the latest keyframe as the center coordinates of each newly generated 3D Gaussian primitive, then optimize the properties of the newly generated and existing 3D Gaussian primitives to obtain the optimized 3D Gaussian scene, and proceed to S5.

[0012] S4: Determine whether the current RGB video frame is the last frame. If yes, obtain the camera trajectory composed of the camera poses corresponding to all RGB video frames. If no, use the next frame of the RGB video as the current RGB video frame and execute S1 to S2.

[0013] S5: Determine if the current RGB video frame is the last frame. If yes, the optimized 3D Gaussian scene is the final 3D Gaussian mapping result; if no, proceed to S6.

[0014] S6: Determine if the number of optimizations is the set number. If not, use the next frame of the RGB video as the current RGB video frame to execute S1 to S2. If yes, in the optimized 3D Gaussian scene, perform densification and pruning on each 3D Gaussian primitive, and then use the next frame of the RGB video as the current RGB video frame to execute S1 to S2.

[0015] Furthermore, the densification and pruning of each 3D Gaussian primitive in the optimized 3D Gaussian scene are specifically performed as follows:

[0016] S51: Divide the 3D Gaussian scene into multiple sub-regions, determine whether the 3D Gaussian primitive density in each sub-region is greater than a set value. For sub-regions with a yes result, the number of primitives remains unchanged. For sub-regions with a no result, increase the number of 3D Gaussian primitives contained therein by cloning and splitting to complete densification.

[0017] S52: Obtain the opacity of all 3D Gaussian primitives in the 3D Gaussian scene, and render the 3D Gaussian scene to obtain the global saliency score of each 3D Gaussian primitive.

[0018] S53: Remove 3D Gaussian primitives whose opacity is below the set threshold T1 and whose global saliency score is below the set threshold T2. The remaining 3D Gaussian primitives are retained in the 3D Gaussian scene to complete the pruning process.

[0019] Furthermore, the method for determining the threshold T2 is as follows:

[0020] The global saliency scores of all 3D Gaussian units are sorted from high to low, and the threshold values ​​are calculated according to a set ratio to obtain the set threshold.

[0021] Furthermore, in step S2, if the judgment result is yes, and the number of all keyframes exceeds the maximum length of the keyframe window after the current RGB video frame is included in the keyframe window as the latest keyframe, then the earliest keyframe in the keyframe window is removed.

[0022] Furthermore, the properties of a 3D Gaussian primitive include its center coordinates, a covariance matrix that controls the primitive's shape, opacity, and spherical harmonic coefficients used to calculate the primitive's color.

[0023] Furthermore, the keyframe initialization method is as follows:

[0024] Set the length of the keyframe window to L;

[0025] The camera pose corresponding to the first RGB video frame in the RGB video is directly defined as the center of the world coordinate system, and the camera pose corresponding to the RGB video frame in the subsequent L-1 frames is calculated by the parallax between the RGB video frame in the subsequent L-1 frames and the first RGB video frame.

[0026] All the first L RGB video frames in the RGB video are included in the keyframe window as keyframes, and the coordinates of each pixel position in the world coordinates on all keyframes are used as the center coordinates of each initially generated 3D Gaussian primitive. At the same time, the initial values ​​of the attributes of each initially generated 3D Gaussian primitive are directly set, or the attributes of each initially generated 3D Gaussian primitive are determined by using an activation function.

[0027] Further, in step S1, the co-visibility between the current RGB video frame and the latest keyframe in the keyframe window is obtained based on the camera pose corresponding to the current RGB video frame.

[0028] Furthermore, in step S2, the DPT-Hybrid model is used to perform monocular depth estimation on the latest keyframe within the keyframe window.

[0029] Furthermore, in step S3, the properties of all newly generated and existing 3D Gaussian primitives are optimized using a combination of photometric loss and regularization loss.

[0030] Furthermore, in steps S4 and S5, it is determined whether the current RGB video frame is the last frame of the RGB video. Beneficial effects:

[0031] This application provides a 3D Gaussian sputtering monocular visual SLAM method that combines neural networks. The method uses neural networks to perform monocular depth estimation on selected keyframes and uses the depth results to optimize 3D Gaussian primitives. At the same time, this application introduces a global saliency score to measure the importance of primitives in a 3D Gaussian scene. The global saliency score and opacity are used as a comprehensive judgment standard for the importance of 3D Gaussian primitives. By making a comprehensive judgment, rendering details are preserved to obtain higher quality rendering results. Attached Figure Description

[0032] Figure 1 is a flowchart of a 3D Gaussian sputtering monocular vision SLAM method combined with neural networks provided in this application;

[0033] Figure 2 is a schematic diagram of the DPT model provided in this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0035] As shown in Figure 1, a 3D Gaussian sputtering monocular vision SLAM method combining neural networks includes the following steps:

[0036] S0: Initialize keyframes:

[0037] Set the length of the keyframe window to L;

[0038] The camera pose corresponding to the first RGB video frame in the RGB video is directly defined as the center of the world coordinate system, and the camera pose corresponding to the RGB video frame in the subsequent L-1 frames is calculated by the parallax between the RGB video frame in the subsequent L-1 frames and the first RGB video frame.

[0039] All the first L RGB video frames in the RGB video are included in the keyframe window as keyframes, and the coordinates of each pixel position in the world coordinates on all keyframes are used as the center coordinates of each initially generated 3D Gaussian primitive. At the same time, the initial values ​​of the attributes of each initially generated 3D Gaussian primitive are directly set, or the attributes of each initially generated 3D Gaussian primitive are determined by using an activation function.

[0040] It should be noted that maps built based on 3D Gaussian primitives can produce better rendering effects. 3D Gaussian mapping, point clouds, polygon meshes, etc. are all mapping methods; the result of 3D Gaussian mapping is composed of a large number of 3D Gaussian primitives. These 3D Gaussian primitives can be visualized as an ellipsoid, which mainly includes the following attributes: [the center position of the ellipsoid in world coordinates (x,y,z), the shape of the ellipsoid (3x3 covariance), the opacity (1 data point), and the spherical harmonic coefficients representing the color (here, we take the 0th order, a total of 3 coefficients)].

[0041] S1: Obtain the camera pose corresponding to the current RGB video frame, and obtain the co-visibility between the current RGB video frame and the latest keyframe in the keyframe window based on the camera pose;

[0042] It should be noted that the camera pose needs to be calculated for each RGB video frame. First, a camera pose is calculated based on the parallax. Then, the existing mapping faces are used to render the camera pose to obtain a rendering result. At this time, the newly received RGB video frame is the ground truth. By calculating the loss function between the rendering result and the ground truth of the RGB frame, the camera pose of the current frame is considered to be optimized when it reaches a minimum value, that is, the camera pose of the current frame is determined.

[0043] S2: Determine whether the co-visibility is lower than the set threshold. If yes, include the current RGB video frame in the keyframe window as the latest keyframe, and use the DPT-Hybrid model to perform monocular depth estimation on the latest keyframe in the keyframe window to obtain the predicted depth value corresponding to each pixel position on the latest keyframe, and then proceed to step S3; if no, proceed to step S4.

[0044] It should be noted that, to further improve system speed, when the judgment result is yes, a new keyframe is generated. If, after including the current RGB video frame in the keyframe window as the latest keyframe, the total number of keyframes exceeds the maximum length of the keyframe window, the oldest keyframe in the keyframe window is discarded. For newly arrived keyframes, monocular depth estimation is performed. The pixel position in the camera coordinate system, combined with the corresponding predicted depth information, undergoes coordinate transformation to become a point (x, y, z) in the world coordinate system. This point serves as the initial center point for the new 3D Gaussian primitives. Each pixel in the keyframe is processed in this way, thus generating many new Gaussian primitives.

[0045] S3: Use the coordinates of each pixel position in the latest keyframe in world coordinates as the center coordinates of each newly generated 3D Gaussian primitive. Then, use the photometric loss and regularization loss to optimize the properties of all newly generated and original 3D Gaussian primitives to obtain the optimized 3D Gaussian scene. Then proceed to step S5.

[0046] Specifically, the attributes of the 3D Gaussian primitives in this application include center coordinates, a covariance matrix controlling the primitive shape, opacity, and spherical harmonic function coefficients used to calculate the primitive color. Because including all video frames in the optimization process would result in very slow computation and little improvement in performance, key frames are identified from all video frames and defined as keyframes. Optimization of the mapping is performed using only keyframes.

[0047] In other words, during the initialization of the map and the subsequent operation, all selected keyframes will undergo monocular depth estimation. The pixel position in the camera coordinate system, combined with the corresponding predicted depth information, will be transformed into a point (x, y, z) in the world coordinate system. This point will serve as the initialization center point of the new 3D Gaussian primitive.

[0048] S4: Determine whether the current RGB video frame is the last frame of the RGB video. If yes, obtain the camera trajectory composed of the camera poses corresponding to all RGB video frames. If no, use the next frame of the RGB video as the current RGB video frame and execute steps S1 to S2.

[0049] In other words, camera pose tracking is performed on each video frame in the input RGB video to obtain a trajectory. In terms of time, there is a one-to-one correspondence between each camera pose and each RGB video frame; that is, one video frame corresponds to one camera pose. The video frames are played back from beginning to end to form a video, and the camera trajectory forms a corresponding trajectory.

[0050] S5: Determine whether the current RGB video frame is the last frame of the RGB video. If yes, the optimized 3D Gaussian scene is the final 3D Gaussian mapping result. If no, proceed to step S6.

[0051] S6: Determine whether the number of optimizations is the set number. If not, directly use the next frame of the RGB video as the current RGB video frame to execute steps S1 to S2. If yes, perform densification and pruning on each 3D Gaussian primitive in the optimized 3D Gaussian scene, and then use the next frame of the RGB video as the current RGB video frame to execute steps S1 to S2.

[0052] It should be noted that densification and pruning are triggered once every fixed number of iterations. Note that this number of iterations starts accumulating after initialization and does not start from 0 every time a new keyframe comes in.

[0053] Specifically, the densification and pruning of each 3D Gaussian primitive in the optimized 3D Gaussian scene are performed as follows:

[0054] S51: Divide the 3D Gaussian scene into multiple sub-regions, determine whether the 3D Gaussian primitive density in each sub-region is greater than a set value. For sub-regions with a yes result, the number of primitives remains unchanged. For sub-regions with a no result, increase the number of 3D Gaussian primitives contained therein by cloning and splitting to complete densification.

[0055] S52: Obtain the opacity of all 3D Gaussian primitives in the 3D Gaussian scene, and render the 3D Gaussian scene to obtain the global saliency score of each 3D Gaussian primitive.

[0056] S53: Remove 3D Gaussian primitives whose opacity is below the set threshold T1 and whose global saliency score is below the set threshold T2. The remaining 3D Gaussian primitives are retained in the 3D Gaussian scene to complete the pruning process.

[0057] Furthermore, the method for determining the threshold T2 is as follows:

[0058] The global saliency scores of all 3D Gaussian units are sorted from high to low, and the threshold values ​​are calculated according to a set ratio to obtain the set threshold.

[0059] The basic principles of monocular depth estimation are introduced below.

[0060] A major drawback of monocular visual SLAM is the lack of depth information, which can be effectively compensated for by introducing monocular depth estimation networks. This application employs the DPT-Hybrid model for monocular depth estimation of keyframes. As shown in Figure 2, the DPT model first extracts non-overlapping features, flattens them, and performs linear projection (DPT-Base and DPT-Large) or converts them into tokens by applying a ResNet-50 feature extractor (DPT-Hybrid). Then, it enhances the image embedding using positional embedding and adds a feature-independent output header. After the tokens pass through multiple Transformer stages, the Reassemble module reassembles the tokens from different stages into image-like representations with multiple resolutions. The Fusion module performs progressive fusion and upsampling to obtain a higher-resolution depth prediction result. As shown in the middle of Figure 2, the Reassemble module reassembles the tokens into a feature map with a spatial resolution of 1 / s of the input image. As shown in the right of Figure 2, the Fusion module uses residual convolutional units to perform feature fusion and upsampling of the feature map.

[0061] The DPT model proposes using Transformer instead of Convolutional Neural Networks as the backbone architecture for dense prediction tasks. The Transformer backbone can process features at relatively stable high resolution, and each stage has a global receptive field. Compared to monocular depth estimation networks using Convolutional Neural Networks or similar structures as the main architecture, the DPT model produces more refined and consistent depth prediction results. DPT-Large and DPT-Hybrid show better performance than DPT-Base, but due to the real-time requirements of visual SLAM, this application uses DPT-Hybrid for monocular depth estimation, which has fewer parameters and lower computational cost.

[0062] It should be noted that the pre-trained parameters of the monocular depth estimation network and dataset can be replaced by other types of neural networks or pre-trained parameters. Different monocular depth estimation networks and pre-trained results should be adopted for different application scenarios to ensure the flexibility and scalability of the technical solution.

[0063] The following section introduces the basic principles of global significance scores and pruning.

[0064] The 3D Gaussian sputtering method proposes a 3D Gaussian primitive pruning strategy based on the primitive's opacity property. This involves setting a threshold and pruning 3D Gaussian primitives with opacities below that threshold. LightGaussian proposes a new criterion for measuring the importance of 3D Gaussian primitives—the global saliency score. After calculating the global saliency scores of all 3D Gaussian primitives, they are sorted and a threshold is calculated based on a certain proportion. 3D Gaussian primitives with global saliency scores below the threshold are then pruned, resulting in a more detailed representation of the scene.

[0065] For a specific primitive, the global significance score is calculated as follows:

[0066] Where j is the Gaussian index, i represents the rendered pixel, and M, H, and W represent the number of views, image height, and width participating in the global saliency score calculation, respectively. 1 is an indicator function determining whether primitive j participated in the rendering of pixel i. Because larger 3D Gaussian primitives participate in the rendering of more pixels, their importance is often exaggerated, leading to the pruning of 3D Gaussian primitives representing scene details. Therefore, in the formula γ(∑ j This represents an adaptive method for restricting the global saliency of large-volume 3D Gaussian elements, and its calculation method is as follows:

[0067] Normalization is performed using 90% of the largest volume among all 3D Gaussian primitives as the standard, and primitives exceeding 1 are restricted to 1 to reduce the impact of primitive volume on global saliency calculation. β is introduced to provide additional flexibility for different application scenarios.

[0068] The global saliency score comprehensively considers factors such as whether 3D Gaussian primitives participate in pixel rendering, opacity, and primitive volume. Because these factors are related to 3D Gaussian scene rendering, a pruning strategy based on the global saliency score is more reasonable than judging solely based on opacity.

[0069] Finally, this application employs a DPT-Hybrid monocular depth model pre-trained on the OmniData dataset for depth prediction. To ensure the running speed of 3DGS monocular visual SLAM, this application loads the model during the initialization phase of tracking and only acquires depth information from keyframes instead of all input RGB video frames. This improves tracking accuracy while providing better center point initialization information for 3D Gaussian primitives, reducing the pressure of mapping optimization and improving mapping performance and speed.

[0070] The global saliency score pruning strategy is applied during the mapping optimization process. When redundant 3D Gaussian primitives appear in the scene, the system will initiate an additional rendering process to calculate the global saliency score and perform pruning. However, because this application uses keyframe windows for optimization, the camera poses corresponding to the windows cannot adequately cover most viewpoints. Therefore, the global saliency scores calculated for the same primitive at different times fluctuate significantly, resulting in a serious map forgetting problem. To improve stability, when the opacity of a 3D Gaussian primitive exceeds a threshold, the primitive will not be pruned.

[0071] In summary, this application provides a 3D Gaussian sputtering monocular visual SLAM method combined with neural networks. After receiving RGB video frames, camera pose estimation is performed, and the co-visibility between the current frame and the previous keyframe is calculated. When the co-visibility is below a threshold, the video frame is selected as a new keyframe, and a DPT model is used for monocular depth estimation. Based on the predicted depth, 3D Gaussian compaction is performed, and the center point positions of the newly generated 3D Gaussian primitives are initialized according to the depth information. When pruning conditions are met, the global saliency scores of all 3D Gaussian primitives in the current scene are calculated and sorted. Primitives ranked lower and whose transparency does not meet the requirements are pruned. After completing the density adjustment and center position initialization of primitives in the scene, a new rendering is performed. The mapping optimization stage calculates the photometric loss and regularization between the current rendering and the previous rendering based on the keyframe currently within the optimization window, optimizing the corresponding camera pose and the attributes of the 3D Gaussian primitives in the scene.

[0072] Therefore, this application addresses the lack of depth information in 3D Gaussian sputtering monocular visual SLAM by introducing a pre-trained monocular depth estimation model to predict the depth information of keyframes. This provides more information for camera pose tracking while minimizing the impact on running speed. Simultaneously, the depth information also allows newly emerging primitives to obtain better center point initialization during 3D Gaussian scene densification, reducing the computational load and time required for map building optimization.

[0073] Furthermore, this application employs a more reasonable 3D Gaussian scene pruning strategy during the mapping optimization process and resolves the instability issues arising from system coupling. By calculating the global saliency score of all 3D Gaussian primitives and comprehensively considering the number of times primitives participate in the calculation, opacity, and primitive volume, the importance of primitives is measured with a more reasonable standard, which can better preserve 3D Gaussian primitives representing details in the scene. Simultaneously, to adapt to the fluctuations in global saliency scores caused by keyframe window mapping optimization, 3D Gaussian primitives whose opacity reaches a threshold during the optimization process will not participate in pruning.

[0074] Of course, there may be other embodiments of this application. Without departing from the spirit and essence of this application, those skilled in the art can make various corresponding changes and modifications based on this application. However, these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A 3D Gaussian sputtering monocular vision SLAM method incorporating neural networks, characterized in that, include: S1: Obtain the co-visibility between the current RGB video frame and the latest keyframe in the keyframe window; S2: Determine if the co-visibility is below the threshold. If yes, include the current RGB video frame in the keyframe window as the latest keyframe, and perform monocular depth estimation on the latest keyframe to obtain the predicted depth value corresponding to each pixel position on the latest keyframe before proceeding to S3; if no, proceed to S4. S3: Use the pixel positions on the latest keyframe as the center coordinates of each newly generated 3D Gaussian primitive, then optimize the properties of the newly generated and existing 3D Gaussian primitives to obtain the optimized 3D Gaussian scene, and proceed to S5. S4: Determine whether the current RGB video frame is the last frame. If yes, obtain the camera trajectory composed of the camera poses corresponding to all RGB video frames. If no, use the next frame of the RGB video as the current RGB video frame and execute S1 to S2. S5: Determine if the current RGB video frame is the last frame. If yes, the optimized 3D Gaussian scene is the final 3D Gaussian mapping result; if no, proceed to S6. S6: Determine if the number of optimizations is the set number. If not, use the next frame of the RGB video as the current RGB video frame to execute S1 to S2. If yes, in the optimized 3D Gaussian scene, perform densification and pruning on each 3D Gaussian primitive, and then use the next frame of the RGB video as the current RGB video frame to execute S1 to S2.

2. The 3D Gaussian sputtering monocular vision SLAM method combined with neural networks as described in claim 1, characterized in that, The specific steps for densification and pruning of each 3D Gaussian primitive in the optimized 3D Gaussian scene are as follows: S51: Divide the 3D Gaussian scene into multiple sub-regions, determine whether the 3D Gaussian primitive density in each sub-region is greater than a set value. For sub-regions with a yes result, the number of primitives remains unchanged. For sub-regions with a no result, increase the number of 3D Gaussian primitives contained therein by cloning and splitting to complete densification. S52: Obtain the opacity of all 3D Gaussian primitives in the 3D Gaussian scene, and render the 3D Gaussian scene to obtain the global saliency score of each 3D Gaussian primitive. S53: Remove 3D Gaussian primitives whose opacity is below the set threshold T1 and whose global saliency score is below the set threshold T2. The remaining 3D Gaussian primitives are retained in the 3D Gaussian scene to complete the pruning process.

3. The 3D Gaussian sputtering monocular vision SLAM method combined with neural networks as described in claim 2, characterized in that, The method for determining the threshold T2 is as follows: The global saliency scores of all 3D Gaussian units are sorted from high to low, and the threshold values ​​are calculated according to a set ratio to obtain the set threshold.

4. The 3D Gaussian sputtering monocular vision SLAM method combined with neural networks as described in claim 1, characterized in that, In step S2, if the judgment result is yes, and the number of all keyframes exceeds the maximum length of the keyframe window after the current RGB video frame is included in the keyframe window as the latest keyframe, then the earliest keyframe in the keyframe window is removed.

5. The 3D Gaussian sputtering monocular vision SLAM method combined with neural networks as described in claim 1, characterized in that, The properties of a 3D Gaussian primitive include its center coordinates, a covariance matrix that controls the primitive's shape, opacity, and spherical harmonic coefficients used to calculate the primitive's color.

6. The 3D Gaussian sputtering monocular vision SLAM method combined with neural networks as described in claim 1, characterized in that, The keyframe initialization method is as follows: Set the length of the keyframe window to L; The camera pose corresponding to the first RGB video frame in the RGB video is directly defined as the center of the world coordinate system, and the camera pose corresponding to the RGB video frame in the subsequent L-1 frames is calculated by the parallax between the RGB video frame in the subsequent L-1 frames and the first RGB video frame. All the first L RGB video frames in the RGB video are included in the keyframe window as keyframes, and the coordinates of each pixel position in the world coordinates on all keyframes are used as the center coordinates of each initially generated 3D Gaussian primitive. At the same time, the initial values ​​of the attributes of each initially generated 3D Gaussian primitive are directly set, or the attributes of each initially generated 3D Gaussian primitive are determined by using an activation function.

7. The 3D Gaussian sputtering monocular vision SLAM method combined with neural networks as described in claim 1, characterized in that, In step S1, the co-visibility between the current RGB video frame and the latest keyframe in the keyframe window is obtained based on the camera pose corresponding to the current RGB video frame.

8. The 3D Gaussian sputtering monocular vision SLAM method combined with neural networks as described in claim 1, characterized in that, In step S2, the DPT-Hybrid model is used to perform monocular depth estimation on the latest keyframe within the keyframe window.

9. The 3D Gaussian sputtering monocular vision SLAM method combined with neural networks as described in claim 1, characterized in that, In step S3, the properties of all newly generated and existing 3D Gaussian primitives are optimized using a combination of photometric loss and regularization loss.

10. The 3D Gaussian sputtering monocular vision SLAM method combined with neural networks as described in claim 1, characterized in that, In steps S4 and S5, it is determined whether the current RGB video frame is the last frame of the RGB video.

Images