Sky Background Modeling for Low-Compute AR Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Augmented-reality (AR) devices face challenges in rendering content far from the user due to the need for large virtual models and high computational resources, especially when using computer-vision machine-learning models for image segmentation in videos.
Innovation Solution
An AR client device generates a background model to identify sky portions in images, using a background detection model to map images onto a 3D structure representing where the sky is visible, reducing the need for extensive virtual modeling and computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If computer-vision machine-learning models are used to segment images and predict AR content presentation at a distance, then AR content can be rendered far from the user, but computational resources required increase significantly
Solution Approach 1:
The patent segments the complex scene into two distinct components: foreground objects (modeled with detailed virtual models) and background sky regions (handled through image segmentation and masking). This segmentation allows the system to apply different processing strategies to different parts of the scene, using lightweight operations for sky regions while maintaining detailed modeling only where necessary, thereby significantly reducing overall computational requirements.
Solution Approach 2:
The patent extracts and isolates the sky background from the full image using machine learning-based segmentation models. By separating the sky portion from foreground objects, the system can process and render AR content for distant sky regions independently from foreground objects, reducing the computational burden of processing entire high-resolution images and enabling efficient distant AR rendering.
2Measurement precision
If all physical objects between the AR device and distant AR content location are modeled, then accurate AR rendering is achieved, but the virtual model size becomes prohibitively large
Solution Approach 1:
The patent applies local quality by differentiating between foreground and background regions and applying appropriate modeling strategies to each. Foreground objects receive detailed virtual model representation for high accuracy, while background sky regions use simplified segmentation-based handling. This localized approach maintains rendering accuracy where needed while avoiding unnecessary complexity in distant sky regions.
Solution Approach 2:
The patent uses image segmentation to create a simplified copy or representation of the sky background rather than creating detailed virtual models of all physical objects in the scene. The segmentation model generates a mask that copies the sky region from the input image, which can then be used for AR content compositing without requiring exhaustive modeling of intermediate objects.
Data Source
AI summary
An AR client device generates and uses a background model to identify portions of images that depict the sky. A background model is a model that represents where the sky is visible for the client device. To identify a sky background portion of an image, a client device can map an image onto the background model and thereby determine which portion of the image represents the sky. The client device can use the identified sky background portion to augment the image to include AR content in the sky. To generate the background model, the client device applies a background detection model to a set of images to generate background probability images. The background probability images are mapped onto a background model using orientation data captured by the client device to update the background model based on the background probability image.