Stereoscopic DNN Depth Estimation Using Semi-Supervised Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional depth estimation methods using deep neural networks (DNNs) face inaccuracies and increased computing requirements due to reliance on monocular images, sparse LIDAR data, and complex network architectures, leading to inefficiencies in real-time deployment, especially in safety-critical applications like autonomous vehicles.
Innovation Solution
The use of semi-supervised training with both LIDAR data and photometric error, combined with a stereoscopic DNN architecture that includes exponential linear unit (ELU) activation functions and a machine learned argmax function, reduces network size and processing requirements while improving accuracy through an encoder/decoder bottleneck architecture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If monocular images are used for depth estimation, then the system is simpler and requires less computing power, but depth accuracy deteriorates due to lack of multiple reference frames
Solution Approach 1:
The patent combines monocular and stereoscopic image inputs into a unified deep neural network architecture. The network processes both single-view and stereo-pair images simultaneously, merging the simplicity of monocular processing with the accuracy benefits of stereoscopic vision to achieve robust depth estimation across varying computational resources
Solution Approach 2:
The deep neural network is designed with universal functionality to handle multiple input types (monocular and stereoscopic images) and multiple output modes (depth maps, disparity maps, segmentation). This multi-functional architecture allows the same system to adapt to different deployment scenarios and computational requirements while maintaining high depth estimation accuracy
2Measurement precision
If DNN is trained on monocular image data for specific geographic regions, then it can identify local vehicles and behaviors accurately, but it requires separate training for each region increasing computing requirements
Solution Approach 1:
The deep neural network employs transfer learning and domain adaptation techniques to achieve universal performance across different geographic regions. The network is trained on diverse datasets from multiple regions and can adapt to local characteristics through fine-tuning or domain adaptation, eliminating the need for complete retraining while maintaining high detection accuracy for region-specific vehicles and behaviors
3Device complexity
If only sparse LIDAR data is used to train the DNN, then the training process is simpler, but the output becomes inaccurate and noisy missing detections
Solution Approach 1:
The patent merges multiple training data sources including sparse LIDAR data, dense photometric data from stereo cameras, and synthetic training data. This combination allows the network to learn from the structured depth information in LIDAR while compensating for sparsity using photometric constraints and synthetic examples, achieving accurate depth estimation without requiring complex training pipelines
Solution Approach 2:
The patent introduces photometric consistency loss and depth supervision as intermediary training mechanisms that bridge the gap between sparse LIDAR data and dense depth prediction. These intermediary signals guide the network to infer accurate depth from limited LIDAR points while maintaining consistency with stereo image observations, reducing output noise and improving detection completeness
4Measurement precision
If conventional stereoscopic DNN uses ReLU activation functions with batch normalization and fully 3D convolutions, then the network can capture spatial features, but the network size increases processing requirements and run-time
Solution Approach 1:
The network architecture segments the feature extraction process into distinct stages: early 3D convolutional layers for spatial feature capture, followed by 2D convolutional layers for refined processing. This segmentation allows the network to benefit from 3D spatial understanding where needed while using computationally efficient 2D operations elsewhere, reducing overall processing requirements while maintaining feature extraction capability
Solution Approach 2:
The patent employs parameter changes by dynamically adjusting convolution kernel sizes, stride values, and activation function selections based on the processing stage and input characteristics. The network uses ELU activation functions with selective batch normalization and adapts 3D vs 2D convolution usage based on computational budget, optimizing the balance between feature extraction quality and processing speed for real-time deployment
Data Source
AI summary
Various examples of the present disclosure include a stereoscopic deep neural network (DNN) that produces accurate and reliable results in real-time. Both LIDAR data (supervised training) and photometric error (unsupervised training) may be used to train the DNN in a semi-supervised manner. The stereoscopic DNN may use an exponential linear unit (ELU) activation function to increase processing speeds, as well as a machine learned argmax function that may include a plurality of convolutional layers having trainable parameters to account for context. The stereoscopic DNN may further include layers having an encoder/decoder architecture, where the encoder portion of the layers may include a combination of three-dimensional convolutional layers followed by two-dimensional convolutional layers.