Stereo 3D Object Detection with Single-Backbone Feature Unmixing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for 3D object detection using stereo cameras face challenges in computation complexity and efficiency, particularly when compared to monocular 2D object detection, and existing deep learning stereo depth estimation methods achieve good accuracy but at the cost of low efficiency due to the use of many 3D convolutional layers.
Innovation Solution
The proposed method employs a Share Network with Feature Unmixing (SNFU) that generates fused images from a pair of stereo images, extracts fused features using a single backbone network, and then unmixes these features into left and right view-aware features using a feature unmixing sub-network. This approach reduces computation complexity and improves efficiency while maintaining high precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If stereo cameras are used for 3D object detection, then depth information accuracy is improved, but computation complexity doubles compared to monocular 2D detection
Solution Approach 1:
The patent merges the processing of left and right view images by concatenating their feature maps along the channel dimension to create fused images. This allows a single backbone network to process both views simultaneously, reducing computation complexity while maintaining the depth information accuracy provided by stereo cameras.
Solution Approach 2:
The single backbone network is designed to handle multiple functions: it extracts features from both left and right view images, performs fusion through channel concatenation, and enables subsequent unmixing operations. This multi-functional design eliminates the need for separate processing pipelines for each view, reducing overall computational burden.
2Measurement precision
If existing deep learning stereo depth estimation methods are used, then accuracy is improved, but efficiency deteriorates due to many 3D convolutional layers
Solution Approach 1:
The patent segments the feature processing into distinct stages: initial feature extraction from stereo images, fusion through channel concatenation, and unmixing into view-specific features. This segmentation avoids the need for numerous 3D convolutional layers by using a more efficient architecture that processes stereo information through fusion and unmixing operations.
Solution Approach 2:
The feature unmixing module acts as an intermediary that separates the fused features back into left and right view-aware features. This intermediary step enables accurate depth estimation without requiring multiple 3D convolutional layers, as the unmixing operation efficiently extracts view-specific information from the fused representation.
3Device complexity
If a single backbone network with feature unmixing is used, then computation complexity is reduced, but the ability to maintain view-specific features may deteriorate
Solution Approach 1:
The feature unmixing module provides feedback to the detection process by separating fused features back into view-specific left and right features. This feedback mechanism ensures that view-specific information is preserved and can be used for accurate 3D object detection, counteracting any potential information loss from using a single backbone network.
Solution Approach 2:
The patent changes the parameter representation of features by concatenating channels from both views to create fused images, then using unmixing to transform these fused features back into view-specific representations. This parameter transformation allows the system to maintain view-specific features while using a computationally efficient single backbone network.
Data Source
AI summary
A method and device for 3D object detection. The method comprises the steps of: generating one or more fused images(201) based on a pair of images(530), the pair of images (530) including a left view image(101,710a) and a right view image(102,710b); extracting one or more fused features from the fused images(201) by a single backbone network(210) of a share network with feature unmixing (SNFU,540); unmixing the fused features to a left view-aware feature and a right view-aware feature by a feature unmixing sub-network(220) of the SNFU(540); predicting the 3D object based on the left view-aware feature and the right view-aware feature; and determining spatial features of the predicted 3D object. The proposed method and device can reduce the computation complexity to the level of that in monocular based 2D object detection so as to improve the computation efficiency while keeping high precision. In addition, the PCNet network for depth estimation can achieve the competitive performance on accuracy and high speed simultaneously when predicting the depth of the 3D object.