Stereo-LiDAR Depth Fusion for Dense Head-Mounted Display Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image generation technologies, such as stereo disparity-based systems and LiDAR-based systems, struggle to produce accurate and dense depth maps, especially in environments with non-textured areas, leading to sparse and noisy results.
Innovation Solution
A method using a neural network with inter-branch weight sharing mechanism, processing first and second colour images and a sparse depth map to generate dense depth maps, integrating information from different branches to enhance depth estimation accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If stereo disparity-based systems are used to generate depth maps, then dense depth maps can be produced in textured areas, but the results become sparse and noisy in non-textured areas
Solution Approach 1:
The patent combines stereo disparity-based systems and LiDAR-based systems into a unified depth estimation framework. The neural network integrates both stereo images and LiDAR depth maps as inputs, merging the dense coverage of stereo vision in textured areas with the reliable performance of LiDAR in non-textured areas, thereby resolving the contradiction between depth map accuracy and completeness
Solution Approach 2:
The patent introduces a neural network as an intermediary that processes and fuses information from multiple sources (stereo images and LiDAR depth maps). This neural network acts as a mediator that reconciles the strengths and weaknesses of each system, producing a unified dense depth map that maintains accuracy across both textured and non-textured surfaces
2Measurement precision
If LiDAR-based systems are used to provide depth information for non-textured areas, then accurate depth information is obtained, but the resulting depth maps become sparse and noisy
Solution Approach 1:
The patent merges LiDAR depth information with stereo image data through a neural network. The LiDAR provides accurate depth measurements for non-textured areas, while the stereo images contribute texture and visual context. The neural network fuses these inputs to generate dense depth maps that maintain the accuracy of LiDAR while achieving the density of stereo vision
Solution Approach 2:
The patent creates a composite depth representation by combining data from heterogeneous sources (stereo images and LiDAR depth maps). This composite approach leverages the complementary strengths of each data source, producing a depth map that is both dense and accurate across diverse surface conditions
3Device complexity
If conventional stereo disparity-based systems or single-camera systems are used, then device complexity is reduced, but the ability to produce accurate dense depth maps for 3D reconstruction is insufficient
Solution Approach 1:
The patent designs a multi-functional system where the head-mounted display device performs both stereo vision and LiDAR depth sensing, and the neural network handles multiple tasks including fusion of heterogeneous data, depth estimation, and super-resolution. This universal approach enables accurate dense depth map generation while maintaining a unified device architecture
Data Source
AI summary
Disclosed is a method for depth estimation including receiving first and second colour images and sparse depth map corresponding to first and second colour images; generating first and second depth prediction maps by providing first and second colour images and sparse depth map as input to neural network, wherein neural network has first branch for receiving first colour image and sparse depth map as input to generate first depth prediction map and second branch for receiving second colour image and sparse depth map as input to generate second depth prediction map, and wherein first and second branches of neural network share weights by inter-branch weight sharing mechanism; and fusing first and second depth prediction maps to generate dense depth map for depth estimation.