Depth Image Completion Using Dual Branch Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current depth image complementing methods face issues such as blurry outputs, unclear edges, and insufficient texture restoration due to simple feature fusion and reliance on manual feature extraction, and lack universality and robustness in handling various depth completion tasks.
Innovation Solution
A deep learning-based method using two branching neural networks, a color branching network, and a depth branching network, with a squeeze-and-excitation block and gated convolution, performs end-to-end training to generate high-quality depth images by merging feature information from both networks, avoiding intermediate representations and ensuring rich detail and edge quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a single regression model is used for depth completion, then the method is simple to implement, but the output image becomes blurry with unclear edges
Solution Approach 1:
The patent divides the depth completion task into two separate regression models: a color branch network that processes RGB images and a depth branch network that processes depth images. This segmentation allows each model to specialize in its respective input type, preventing the edge blurring that occurs when a single model attempts to handle both types simultaneously. The color branch extracts features from color images while the depth branch processes depth map features, and their results are fused to produce high-quality output.
2Device complexity
If simple feature fusion is used in deep learning methods, then the network structure remains simple, but the texture restoration and edge quality become insufficient
Solution Approach 1:
The patent combines features from multiple sources through a feature fusion module. Specifically, it merges the color image features extracted by the color branch network with the depth map features from the depth branch network. This merging occurs at multiple levels: early feature fusion combines low-level features, while late feature fusion combines high-level semantic features. This multi-level fusion strategy significantly improves texture restoration and edge quality compared to simple feature concatenation.
Solution Approach 2:
The patent creates a composite feature representation by combining features from different modalities (color and depth) at multiple hierarchical levels. The feature fusion module integrates RGB features and depth features into a composite feature map that contains both texture information from color images and geometric information from depth maps, enabling superior texture restoration and edge preservation in the final depth completion output.
3Adaptability or versatility
If manual feature extraction is used, then the method requires detailed function design, but the development is restricted and lacks universality
Solution Approach 1:
The patent employs self-service through automated feature extraction using convolutional neural networks. Instead of requiring manual design of feature extraction functions, the CNN automatically learns and extracts relevant features from input images during the training process. The color branch and depth branch networks each contain convolutional layers that automatically discover important features specific to their input types, eliminating the need for hand-crafted feature detectors and making the method universally applicable to different datasets without manual adjustment.
4Measurement precision
If RGB image information is used to guide depth completion, then accuracy improves, but the handling of edge portions and large missing depth portions remains unsatisfactory
Solution Approach 1:
The patent adds another dimension to feature extraction by processing RGB images through a separate color branch network that operates in parallel with the depth branch. This color branch extracts rich texture and edge information from the color dimension that is then fused with depth features. By utilizing this additional color information dimension rather than relying solely on depth map interpolation, the method significantly improves edge portion quality and handles large missing depth portions effectively through the complementary color features.
Data Source
AI summary
A method and apparatus for complementing a depth image are provided. The method includes obtaining a color image and a corresponding depth image, obtaining a first depth image based on the color image, using a first deep neural network (DNN), obtaining a second depth image based on the depth image and an intermediate feature image generated by each intermediate layer of the first DNN, using a second DNN, and obtaining a final depth image by merging the first depth image and the second depth image.