Depth Completion Using Spatial-Temporal Fusion for Autonomous Driving
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing depth completion methods for autonomous vehicles face challenges in accuracy due to noise and varying lighting conditions, which can result in incomplete or erroneous depth data from LiDAR sensors.
Innovation Solution
A depth completion method and apparatus utilizing spatial-temporal information, which processes RGB images and sparse LiDAR data through encoder-decoder branches to generate dense depth maps, and further refines these maps using convolutional Long Short-Term Memory (LSTM) to incorporate temporal information and track moving objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If LiDAR is used to acquire depth information, then distance measurement capability is improved, but measurement precision deteriorates due to noise and lighting conditions
Solution Approach 1:
The patent combines LiDAR depth information with RGB image data through a multi-branch neural network architecture. The first branch processes RGB images to generate depth maps, while the second branch processes LiDAR point clouds, and both are fused to produce the final depth completion output, leveraging complementary strengths of both sensors
Solution Approach 2:
The patent introduces a convolutional LSTM network as an intermediary temporal processing layer that integrates historical depth information and current sensor data. This mediator component fuses spatial information from multiple sources with temporal context to compensate for LiDAR measurement errors caused by noise and lighting conditions
2Device complexity
If only spatial information is used for depth completion, then processing complexity is reduced, but depth completion accuracy deteriorates
Solution Approach 1:
The patent employs a convolutional LSTM network that dynamically processes temporal sequences of depth information. The network adapts to changing scene conditions by learning from historical frames, allowing the system to capture temporal dynamics of moving objects and improve depth completion accuracy beyond static spatial processing
Solution Approach 2:
The patent transitions from purely spatial processing to spatio-temporal processing by adding the time dimension. The multi-branch architecture processes spatial features from RGB and LiDAR, while the convolutional LSTM integrates temporal information across video frames, effectively adding another dimension to the processing space
3Measurement precision
If multi-branch encoder-decoder architecture is used, then depth map quality is improved, but device complexity increases
Solution Approach 1:
The patent divides the depth completion task into separate processing branches: a first branch for RGB image processing, a second branch for LiDAR point cloud processing, and a fusion branch for integrating both. Each branch has its own encoder-decoder structure optimized for specific input types, allowing modular processing that improves quality while managing complexity through functional decomposition
Data Source
AI summary
Provided are a depth completion method and apparatus using spatial-temporal information. The depth completion apparatus according to the present invention comprises a processor; and a memory connected to the processor, wherein the memory stores program instructions executable by the processor for performing operations comprising receiving an RGB image and a sparse image through a camera and LiDAR, generating a dense first depth map by processing color information of the RGB image through a first branch based on an encoder-decoder, generating a dense second depth map by up-sampling the sparse image through a second branch based on an encoder-decoder, generating a third depth map by fusing the first depth map and the second depth map, and generating a final depth map including a trajectory of a moving object included in an RGB image continuously captured during movement by inputting the third depth map to a convolution long term short memory (LSTM).