Raw Camera and Depth Sensor Fusion for Dense Depth Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous vehicle systems face challenges in generating accurate and dense depth maps using raw sensor data from camera and depth sensors, which affects perception, planning, and control operations.
Innovation Solution
A neural network-based approach is employed to fuse raw sensor data from camera and depth sensors, such as ultrasonic and ToF cameras, to generate a dense depth map with improved resolution and accuracy, utilizing a Siamese neural network architecture for data fusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If raw sensor data from camera and depth sensors is fused using traditional methods, then the system can provide depth information, but the depth map resolution and accuracy remain insufficient for reliable autonomous vehicle operations
Solution Approach 1:
The patent combines raw data from multiple sensor types (camera, ultrasonic, ToF) into a unified depth map using neural network-based sensor fusion. The system merges the high-resolution visual data with depth measurements to produce an integrated depth map that leverages the complementary strengths of each sensor modality, achieving superior accuracy compared to individual sensors or traditional fusion methods.
Solution Approach 2:
The patent introduces a neural network as an intermediary processing layer between raw sensor data and the final depth map. This neural network mediator learns optimal fusion strategies and transforms heterogeneous sensor inputs into a coherent dense depth map, resolving the complexity of direct sensor integration while maintaining high measurement precision.
2Measurement precision
If dense depth maps with high resolution are generated, then perception accuracy improves, but computational resources and processing time increase
Solution Approach 1:
The patent performs preliminary processing by fusing raw sensor data at the earliest stage in the pipeline, before subsequent perception and planning operations. By generating the dense depth map upfront using neural network fusion, the system avoids repeated heavy computations later, reducing overall processing time while maintaining high resolution output.
Solution Approach 2:
The patent transforms the processing approach by changing from traditional geometric fusion methods to neural network-based fusion. This parameter change in the fusion methodology enables efficient computation of high-resolution depth maps through learned patterns, reducing computational burden compared to exhaustive traditional algorithms while preserving detail.
3Reliability
If multiple sensor types are integrated for data fusion, then measurement reliability improves, but system complexity and calibration requirements increase
Solution Approach 1:
The patent implements self-calibration mechanisms where the neural network automatically adapts to sensor characteristics and relative positioning during operation. The system performs self-service calibration by learning sensor-specific patterns and transformations from training data, eliminating the need for manual calibration procedures and reducing the operational complexity of multi-sensor integration.
Data Source
AI summary
Systems and techniques are provided for fusing raw sensor data captured by a camera sensor and one or more depth sensors to generate a dense depth map. An example process includes receiving raw camera data captured by a camera sensor and descriptive of a scene, receiving raw depth data captured by one or more depth-sensing sensors and descriptive of the scene, and providing the raw camera data and the raw depth data to a neural network, which is configured to fuse the raw camera data and the raw depth data. The example process can further include generating a depth map of the scene based on the fusion of the raw camera data and the raw depth data.