Depth Map Generation Using Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for generating depth maps in computer-generated reality environments, particularly in mixed reality, often rely on sparse depth information, which limits the accuracy and completeness of depth data, affecting the user experience by not fully capturing the physical environment's depth details.
Innovation Solution
The method involves capturing images from multiple perspectives using one or more scene cameras and generating dense depth maps by combining sparse depth maps with pixel values, utilizing techniques such as visual inertial odometry and deep learning neural networks to increase the depth information density, allowing for more precise depth calculations across a wider range of pixels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If sparse depth information is used, then device complexity is reduced, but measurement precision of depth data deteriorates
Solution Approach 1:
The system captures images from multiple perspectives in advance and generates preliminary sparse depth maps before the final depth reconstruction. This preliminary action provides a foundation that can be later enhanced without requiring complex real-time processing during the main depth map generation phase.
Solution Approach 2:
The patent introduces an intermediary deep learning neural network that transforms sparse depth information into dense depth maps. This intermediary component bridges the gap between simple sparse depth data and the desired high-precision dense depth map, allowing the system to maintain low complexity while achieving high measurement precision through the learned transformations.
2Measurement precision
If dense depth maps are generated using multiple perspectives and deep learning, then measurement precision of depth information is improved, but device complexity increases
Solution Approach 1:
The patent replaces complex mechanical depth sensing systems with a computational approach using deep learning neural networks. Instead of using multiple complex depth sensors or sophisticated mechanical scanning systems, the invention uses standard image cameras combined with AI-based processing to achieve dense depth map generation, thereby reducing hardware complexity while maintaining or improving measurement precision.
3Loss of time
If sparse depth maps are used, then processing time is reduced, but loss of information increases
Solution Approach 1:
The system changes the parameter of depth information density by using deep learning to transform sparse depth data into dense depth maps. This parameter change allows the system to maintain fast processing speeds while significantly reducing information loss, as the neural network efficiently infers depth values for pixels that would otherwise be missing in sparse representations.
Data Source
AI summary
In one implementation, a method of generating a depth map is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes generating, based on a first image and a second image, a first depth map of the second image. The method includes generating, based on the first depth map of the second image and pixel values of the second image, a second depth map of the second image.