AR Occlusion Rendering via Depth Map Pixel Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality (AR) systems face challenges in correctly rendering and displaying AR elements to appear occluded or partially occluded by the real-world scene, leading to user confusion, especially in navigation systems where AR elements should be behind objects closer than themselves.
Innovation Solution
A depth map is generated using a depth estimation model to determine the distance of real-world pixels, allowing for accurate occlusion of AR elements by comparing these distances with the AR element's distance, ensuring proper placement and reduced user confusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If AR elements are displayed covering or overlapping the scene, then the AR elements are visible and prominent, but the spatial relationship becomes incorrect causing user confusion
Solution Approach 1:
The patent applies preliminary action by generating a depth map before displaying AR elements. The depth estimation model processes the scene to create depth information in advance, allowing the system to determine occlusion relationships beforehand. This enables correct spatial rendering without requiring complex real-time calculations during display, thus improving spatial relationship accuracy while managing rendering complexity.
2Measurement precision
If depth map processing is performed to achieve accurate occlusion, then spatial relationship accuracy is improved, but processing time increases
Solution Approach 1:
The patent replaces complex mechanical depth sensing systems with a neural network-based depth estimation model. Instead of using multiple cameras or LIDAR hardware to physically measure depth, the system uses a trained neural network to estimate depth from monocular video input. This substitution achieves accurate occlusion rendering while maintaining real-time processing speeds, as the neural network can generate depth maps quickly from standard video feeds.
3Measurement precision
If a depth estimation model is used to generate depth maps, then occlusion rendering accuracy is improved, but computational requirements increase
Solution Approach 1:
The patent uses a neural network model that has been pre-trained and copied to the client device. Instead of performing complex depth estimation calculations from scratch during runtime, the system uses the pre-trained model's learned parameters to rapidly generate depth maps. This copying approach allows accurate depth estimation with reduced computational energy consumption during actual AR rendering, as the heavy lifting was done during the model training phase.
Data Source
AI summary
AR elements are occluded in video image frames. A depth map is determined for an image frame of a video received from a video capture device. An AR graphical element for overlaying over the image frame is received. An element distance for AR graphical elements relative to a position of a user of the video capture device (e.g., the geographic position of the video capture device) is also received. Based on the depth map for the image frame, a pixel distance is determined for each pixel in the image frame. The pixel distances of the pixels in the image frame are compared to the element distance, and in response to a pixel distance for a given pixel being less than the element distance, the pixel of the image frame is displayed rather than a corresponding pixel of the AR graphical element.