Dense Depth Map Reconstruction for Lightweight XR

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Lightweight extended reality (XR) systems face challenges in reconstructing and re-projecting 3D scenes due to reduced processing power, low-resolution cameras, and simplified tracking optics, resulting in blurred or washed-out pixel edges, which impede accurate 3D scene reconstruction and comprehension.

Innovation Solution

The technique involves reconstructing dense depth maps based on sparse depth points and corresponding image frames using a cost function that propagates depth points, determined by static and dynamic feature points, with weights adjusted for depth coherence between neighboring frames, utilizing linear solvers like full multigrid, geometric multigrid, or algebraic multigrid solvers to improve 3D scene reconstruction accuracy and precision.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Weight of moving object

If lightweight XR systems use reduced processing power and low-resolution cameras, then device weight and power consumption are reduced, but 3D scene reconstruction accuracy deteriorates

Engineering Contradiction:

Improvedevice weightVSAvoid3D scene reconstruction accuracy

Core Design Contradiction:

Weight of moving objectVSMeasurement precision

Solution Approach 1:

The system performs preliminary actions by capturing multiple image frames and identifying feature points before depth map reconstruction. Sparse depth points are propagated through multiple frames using cost functions that leverage temporal coherence, allowing the system to build accurate depth information progressively rather than requiring high processing power in real-time

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Feature points serve as intermediaries between the low-resolution camera input and the final depth map output. The cost function uses these feature points as reference markers to propagate depth information across frames, bridging the gap between limited sensor data and accurate 3D reconstruction

Inventive Principle:

Principle #24Intermediary (Mediator)

2Device complexity

If lightweight XR systems use simplified tracking optics, then device complexity is reduced, but depth map quality deteriorates

Engineering Contradiction:

Improvetracking optics complexityVSAvoiddepth map quality

Core Design Contradiction:

Device complexityVSManufacturing precision

Solution Approach 1:

The system transitions from spatial dimension analysis to temporal dimension analysis by propagating depth points across multiple image frames. The cost function incorporates temporal coherence by comparing depth values at the same spatial location across different frames, effectively using the time dimension to compensate for simplified optics

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Solution Approach 2:

The depth reconstruction process maintains continuity by continuously propagating depth points across a sequence of image frames. Rather than processing isolated frames, the system maintains ongoing depth estimation that accumulates information over time, ensuring consistent and high-quality depth maps despite simpler optics

Inventive Principle:

Principle #20Continuity of useful action

3Use of energy by moving object

If lightweight XR systems re-project 3D renderings with reduced processing, then power consumption is reduced, but pixel edge quality deteriorates

Engineering Contradiction:

Improvepower consumptionVSAvoidpixel edge quality

Core Design Contradiction:

Use of energy by moving objectVSMeasurement precision

Solution Approach 1:

The system extracts only the essential depth information needed for accurate re-projection by identifying and propagating key feature points through the cost function. Rather than processing all pixels equally, the system focuses computational resources on determining depth values at critical locations, then interpolates to generate complete depth maps with sharp edges

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The cost function dynamically adjusts parameters based on depth coherence between frames, weighting the contribution of different frames according to their reliability. This adaptive parameter adjustment allows the system to maintain high pixel edge quality by leveraging temporal information when available, while consuming less power by adapting to scene stability

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11615594B2Systems and methods for reconstruction of dense depth maps

Publication Date: 2023.03.28 SAMSUNG ELECTRONICS CO LTD

AI summary

A method by an extended reality (XR) display device includes accessing image data and sparse depth points corresponding to a plurality of image frames to be displayed on one or more displays of the XR display device. The method further includes determining a plurality of sets of feature points for a current image frame of the plurality of image frames, constructing a cost function configured to propagate the sparse depth points corresponding to the current image frame based on the plurality of sets of feature points, and generating a dense depth map corresponding to the current image frame based on an evaluation of the cost function. The method thus includes rendering the current image frame on the one or more displays of the XR display device based on the dense depth map.