Monocular Depth Estimation Using 2D-to-3D Feature Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for monocular depth estimation from a single image often inaccurately represent depth due to undetected occlusions and lack of effective spatial relationship detection.
Innovation Solution
A hybrid neural network architecture combining a two-dimensional convolutional neural network (2D CNN) for feature extraction and a three-dimensional convolutional neural network (3D CNN) to process images, leveraging a linear combination of 2D features to form a 3D input, enabling accurate depth estimation by detecting spatial relationships and occlusions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single two-dimensional image is used for depth estimation, then processing requirements are reduced, but depth accuracy deteriorates due to inability to detect occlusions and spatial relationships
Solution Approach 1:
The patent transforms two-dimensional image features into a three-dimensional feature space by stacking multiple 2D feature maps with different semantic meanings along the channel dimension. This dimensional transformation enables the model to capture spatial relationships and occlusion information that are not apparent in single 2D images, thereby improving depth estimation accuracy while still processing single images.
Solution Approach 2:
The patent combines multiple types of features (edge features, gradient features, semantic features) extracted by different 2D convolutional networks into a composite 3D feature representation. This composite feature structure integrates diverse information sources, enabling the system to detect occlusions and spatial relationships more effectively than single-feature approaches.
2Measurement precision
If hybrid neural network architecture (2D CNN + 3D CNN) is used for depth estimation, then depth accuracy and occlusion detection are improved, but device complexity increases
Solution Approach 1:
The patent divides the depth estimation task into two sequential stages: first, a 2D CNN extracts multiple types of features from the input image; second, a 3D CNN processes the stacked 3D feature representation to estimate depth. This segmentation allows each network component to specialize in specific feature extraction tasks, improving overall accuracy while keeping individual modules manageable in complexity.
Solution Approach 2:
The patent performs preliminary feature extraction using 2D CNNs before feeding data to the 3D CNN. Multiple 2D networks pre-process the image to extract edge, gradient, and semantic features, which are then stacked into a 3D structure. This preliminary action reduces the computational burden on the 3D CNN and enables it to focus on spatial relationship analysis, thereby improving efficiency despite the increased overall architecture complexity.
Data Source
AI summary
A system includes a computing device. The computing device is configured to perform a set of functions. The set of functions includes receiving an image, wherein the image comprises a two-dimensional array of data. The set of functions includes extracting, by a two-dimensional neural network, a plurality of two-dimensional features from the two-dimensional array of data. The set of functions includes generating a linear combination of the plurality of two-dimensional features to form a single three-dimensional input feature. The set of functions includes extracting, by a three-dimensional neural network, a plurality of three-dimensional features from the single three-dimensional input feature. The set of functions includes determining a two-dimensional depth map. The two-dimensional depth map contains depth information corresponding to the plurality of three-dimensional features.