Scene Reconstruction Heightfields for Real-Time AR Occlusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing three-dimensional scene reconstruction methods from input image data are hindered by high runtime and memory requirements, leading to latency in augmented reality applications and agent navigation, which can delay virtual character placement and navigational guidance.
Innovation Solution
A scene reconstruction model that hybridizes the generation of raw and refined heightfields by predicting depth maps and feature maps from input images, using a combination of depth network and feature network to create a 3D representation, allowing for real-time reconstruction of occluded areas by blending the raw and refined heightfields.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-quality scene reconstruction methods are used, then reconstruction quality is improved, but runtime and memory requirements increase causing latency
Solution Approach 1:
The reconstruction process is divided into two separate networks: a depth prediction network that generates depth maps for real-time performance, and a feature map network that provides detailed scene features. This segmentation allows each network to be optimized independently, with the depth network prioritizing speed and the feature network prioritizing quality, thereby resolving the contradiction between reconstruction quality and runtime latency.
Solution Approach 2:
The patent transitions from traditional single-network reconstruction to a multi-dimensional approach by combining depth maps (spatial dimension) with feature maps (semantic dimension). This dimensional expansion allows the system to achieve high-quality reconstruction through complementary information sources rather than relying on a single computationally intensive network, thus improving quality without proportionally increasing runtime.
2Measurement precision
If high-quality scene reconstruction methods are used, then reconstruction quality is improved, but memory requirements increase
Solution Approach 1:
By segmenting the reconstruction task into separate depth and feature networks, each network can be optimized for its specific function with appropriate memory allocation. The depth network uses less memory by focusing only on geometric information, while the feature network uses memory efficiently by processing only relevant semantic features, collectively reducing total memory requirements compared to a monolithic high-quality reconstruction network.
Solution Approach 2:
The patent extracts and separates different types of information (depth vs. features) into distinct processing streams. This extraction allows the system to discard redundant information and process only the essential components needed for quality reconstruction, thereby reducing memory requirements while maintaining or improving reconstruction quality.
3Use of energy by moving object
If delayed scene reconstruction occurs, then processing resources are conserved, but agent navigation guidance is delayed
Solution Approach 1:
The patent implements dynamic processing where the depth prediction network operates continuously at high speed to provide real-time navigation guidance, while the feature map network processes information asynchronously when resources are available. This dynamic architecture ensures that critical navigation functions are never delayed while allowing the system to conserve resources by processing non-critical features only when processing capacity is available.
Data Source
AI summary
A scene reconstruction model is disclosed that outputs a heightfield for a series of input images. The model, for each input image, predicts a depth map and extracts a feature map. The model builds a 3D model utilizing the predicted depth maps and camera poses for the images. The model raycasts the 3D model to determine a raw heightfield for the scene. The model utilizes the raw heightfield to sample features from the feature maps corresponding to positions on the heightfield. The model aggregates the sampled features into an aggregate feature map. The model regresses a refined heightfield based on the aggregate feature map. The model determines the final heightfield based on a combination of the raw heightfield and the refined heightfield. With the final heightfield, a client device may generate virtual content augmented on real-world images captured by the client device.