Stereoscopic Depth Mapping for Real-Time AR/XR Scene Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality (AR) and extended reality (XR) devices face challenges in performing three-dimensional scene reconstruction while maintaining system performance, including frame rate, battery life, and processor usage, due to the computational demands of real-time image processing and rendering.
Innovation Solution
A system and method for depth and scene reconstruction using a stereoscopic camera pair, inertial measurement units (IMUs), and neural networks to generate dense depth maps and three-dimensional reconstructions, optimizing processing loads through image rectification, feature extraction, and pose tracking, and rendering virtual objects on real-world surfaces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If three-dimensional scene reconstruction and dense depth mapping are performed in real-time on portable AR/XR devices, then scene understanding quality is improved, but processor usage and energy consumption increase significantly
Solution Approach 1:
The patent segments the depth mapping process into multiple processing stages: initial coarse depth map generation, iterative refinement passes, and selective processing of different scene regions. This segmentation allows the system to perform comprehensive scene reconstruction while controlling computational load at each stage, thereby reducing overall energy consumption compared to single-pass full-resolution processing.
Solution Approach 2:
The patent implements partial processing by initially mapping only a subset of pixels to depth values, then iteratively refining the depth map for remaining pixels based on scene complexity and processing resources available. This partial action approach ensures adequate scene understanding is achieved without the excessive energy cost of processing every pixel at full resolution simultaneously.
2Productivity
If real-time image processing is performed to maintain high frame rates, then display quality is improved, but processor usage increases
Solution Approach 1:
The patent employs periodic action by performing full-depth-mapping operations at selected frames rather than every frame. Between these full processing cycles, the system uses the previously computed depth maps and performs lighter processing tasks. This periodic approach maintains high frame rates for smooth display while reducing overall processor usage by avoiding redundant full scene reconstruction at every frame.
Solution Approach 2:
The patent performs preliminary action by pre-computing feature maps and disparity information during less demanding periods, then using these pre-computed results to accelerate real-time depth mapping when frame generation is required. This preliminary preparation reduces the computational burden during critical rendering phases, enabling high frame rates with reduced peak processor usage.
3Measurement precision
If dense depth maps are generated through iterative refinement, then depth mapping accuracy is improved, but processing time increases
Solution Approach 1:
The patent implements dynamics by adaptively adjusting the number of iterative refinement passes based on scene characteristics and processing constraints. For simple scenes or time-critical moments, fewer iterations are performed; for complex scenes with more processing budget, additional iterations refine the depth map further. This dynamic adjustment maintains high depth mapping accuracy where needed while reducing processing time in other scenarios.
Solution Approach 2:
The patent applies local quality by performing iterative refinement selectively on specific regions of the depth map rather than uniformly across the entire image. Regions with high scene complexity, occlusions, or importance for AR rendering receive additional refinement iterations, while simpler regions use fewer iterations. This localized approach improves depth mapping accuracy in critical areas without proportionally increasing overall processing time.
Data Source
AI summary
A method includes obtaining first and second image data of a real-world scene, performing feature extraction to obtain first and second feature maps, and performing pose tracking based on at least one of the first image data, second image data, and pose data to obtain a 6DOF pose of an apparatus. The method also includes generating, based on the 6DOF pose, first feature map, and second feature map, a disparity map between the image data and generating an initial depth map based on the disparity map. The method further includes generating a dense depth map based on the initial depth map and a camera model and generating, based on the dense depth map, a three-dimensional reconstruction of at least pail of the scene. In addition, the method includes rendering an AR or XR display that includes one or more virtual objects positioned to contact one or more surfaces of the reconstruction.