ConvLSTM Neural Network for Depth Map Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for acquiring dense depth maps, especially for autonomous vehicles, face challenges in achieving high accuracy and reliability at a reasonable cost, with existing methods using lidars being expensive and camera-based stereovision methods lacking reliability in certain environments.
Innovation Solution
A depth maps prediction system utilizing a neural network with an encoder, ConvLSTM, and decoder sub-networks, which processes images and sparse depth maps to generate dense depth maps, leveraging ConvLSTM's ability to efficiently process spatial data and maintain memory of previous images, thereby improving accuracy and reducing the number of parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high resolution lidars are used to acquire dense depth maps, then measurement precision is improved, but device cost increases significantly
Solution Approach 1:
The patent combines multiple data sources (sparse depth maps from lidars, images from cameras, and temporal information from video sequences) into a unified deep learning framework. This merging allows the system to achieve dense depth map accuracy comparable to high-resolution lidars while using cheaper sensor combinations.
Solution Approach 2:
The system uses deep learning networks to generate synthetic dense depth maps that replicate the output of expensive high-resolution lidars. The neural networks learn to copy the depth estimation capabilities of high-resolution lidars by training on paired data, enabling cost-effective deployment.
2Ease of manufacture
If paired cameras with stereovision are used to acquire depth maps, then device cost is reduced, but measurement precision deteriorates in certain environments
Solution Approach 1:
The patent introduces deep learning networks as intermediary components that process and enhance the raw stereovision data. These networks learn to compensate for the limitations of stereovision in challenging environments by leveraging temporal consistency and sparse depth map guidance, thereby improving accuracy without increasing hardware cost.
Solution Approach 2:
The system creates a composite approach by combining multiple data modalities (stereovision, monocular depth estimation, sparse lidar data, and temporal information) within a unified neural network framework. This composite methodology leverages the strengths of each modality to overcome the weaknesses of individual approaches.
3Measurement precision
If LSTM networks are used to process video sequences for depth prediction, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent extracts and processes only the essential temporal features needed for depth prediction using LSTM networks, rather than processing all raw video data. By extracting key temporal dependencies and feeding them into the depth prediction network, the system maintains accuracy while reducing computational complexity.
Solution Approach 2:
The system segments the processing pipeline into distinct modules: feature extraction from video sequences, temporal processing via LSTM, and final depth prediction. This segmentation allows each module to be optimized independently, reducing overall complexity while maintaining precision.
4Ease of manufacture
If sparse depth maps from low-resolution lidars are used as input, then device cost is reduced, but measurement precision deteriorates
Solution Approach 1:
The system performs preliminary processing of sparse depth maps through the encoder network, which extracts meaningful features and prepares the data for subsequent refinement. This preliminary action transforms the low-quality sparse inputs into enriched feature representations that guide the generation of high-quality dense depth maps.
Solution Approach 2:
The deep learning network transforms the sparsity parameter of the input depth maps by learning to infer missing depth values based on spatial and temporal context. The network changes the density parameter from sparse to dense while maintaining cost-effectiveness by using low-resolution lidar inputs.
Data Source
AI summary
A depth maps prediction system comprising a neural network (1000) configured to receive images (I) of a scene at successive time steps (t−1, t, t+1, . . . ) and comprising three sub-networks: an encoder (100), a ConvLSTM (200) and a decoder (300).The neural network (1000) is configured so that at each time step:a) the encoder sub-network (100) processes an image (I) and outputs a low resolution initial image representation (X);b) the CONVLSTM sub-network (200) processes the initial image representation (X), values for a previous time step (t−1) of an internal state (C(t−1)) and of an LSTM hidden variable data (H(t−1)) of the ConvLSTM sub-network, and outputs updated values of the internal state (C(t)) and of the LSTM hidden variable data (H(t)); andc) the decoder sub-network (300) inputs the LSTM output data (LOD) and generates a predicted dense depth map (D″) for the inputted image (I).