3D Reconstruction Using RGB-Depth Segmentation for Noise Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The presence of noise in collected images of three-dimensional objects, such as non-rigid areas and temporary medical items, affects the accuracy and efficiency of three-dimensional reconstruction processes, particularly in oral medical applications like teeth and gums, due to difficulties in distinguishing between similar textures and lighting issues in RGB images.
Innovation Solution
A method involving preprocessing depth images using structured light, adjusting pixel values, and performing semantic segmentation with combined RGB and depth images to accurately segment target and non-target objects, followed by three-dimensional reconstruction to remove noise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If semantic segmentation is performed only on RGB images, then the process is simple, but the segmentation accuracy is low due to similar textures and lighting issues
Solution Approach 1:
The patent combines RGB images and depth images into a fused image that integrates color information and depth information. This fusion allows the semantic segmentation model to leverage both texture/color cues from RGB and spatial/depth cues from depth images, significantly improving segmentation accuracy for objects with similar textures or under challenging lighting conditions while maintaining reasonable processing complexity through efficient fusion algorithms.
Solution Approach 2:
The patent introduces the depth dimension by incorporating depth images into the semantic segmentation process. Instead of relying solely on the two-dimensional RGB image space, the system adds a third dimension (depth) to create a multi-dimensional input for the segmentation model. This additional dimensional information helps distinguish between objects that appear similar in RGB space but have different spatial positions or depths, thereby improving segmentation accuracy.
2Manufacturing precision
If noise information is not removed from depth images, then the reconstruction process is faster, but the accuracy of the three-dimensional model is lower
Solution Approach 1:
The patent performs noise removal and semantic segmentation on depth images before the three-dimensional reconstruction process. By preprocessing the depth images to eliminate noise information and extract only relevant target object regions, the system ensures high-quality input data for reconstruction. This preliminary action prevents noise from degrading the final model accuracy while the optimized processing pipeline maintains efficient reconstruction speed.
Solution Approach 2:
The patent extracts and removes noise information and non-target objects from depth images through semantic segmentation. By identifying and isolating only the relevant target object regions in the depth images, the system eliminates harmful noise data that would otherwise slow down processing or degrade reconstruction quality. This extraction process ensures that only clean, relevant data proceeds to the reconstruction stage.
3Area of stationary object
If multiple views including noisy areas are used for reconstruction, then more complete coverage is achieved, but the reconstruction efficiency decreases and model accuracy is reduced
Solution Approach 1:
The patent performs semantic segmentation and noise removal on multiple view images before the reconstruction process. By preprocessing each view to identify and retain only relevant target object areas while removing noise and non-target objects, the system ensures that all input views are optimized. This allows the reconstruction algorithm to process multiple views efficiently without being burdened by noise, thereby maintaining both complete coverage and high reconstruction efficiency.
Solution Approach 2:
The patent applies semantic segmentation to divide the scene in each view into target object regions and non-target noise regions. By segmenting multiple views at the pixel level and retaining only the target object portions, the system achieves complete coverage of the target object from all views while eliminating noise. This segmentation approach allows efficient processing of multiple views since the reconstruction algorithm only needs to process the segmented target regions rather than entire images.
Data Source
Figure 1~2
Figure 3~4
Figure 5~7
AI summary
The present application discloses a three-dimensional reconstruction method and apparatus, an electronic device, and a computer-readable storage medium. The three-dimensional reconstruction method comprises: acquiring a first depth image to be reconstructed of a target scene and an RGB image corresponding to said first depth image; preprocessing said first depth image to obtain a second depth image to be reconstructed; on the basis of said second depth image and the RGB image, performing semantic segmentation on said second depth image, and determining a target object area and a non-target object area in said second depth image; processing said second depth image to remove the non-target object area in said second depth image; and performing three-dimensional reconstruction by using said processed second depth image to obtain a three-dimensional model of a target object. The accuracy of semantic segmentation in the three-dimensional reconstruction process of a target object can be improved, thereby improving the efficiency of three-dimensional reconstruction of the target object and the accuracy of the reconstructed three-dimensional model.