Pixel Depth Determination for Augmented Reality Object Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality (AR) systems struggle to accurately replace backgrounds in images or videos without using depth sensors, leading to poor image quality and incorrect identification of objects, especially when objects are partially or fully outside the camera's focus.
Innovation Solution
The system employs a machine learning model to estimate the depth of a point of interest and the distance between each pixel and the point of interest, allowing for a dense depth reconstruction of the object. This reconstruction enables the application of AR effects to the object without affecting the background.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth sensors are used to accurately replace backgrounds in AR systems, then background replacement accuracy is improved, but device complexity and cost increase
Solution Approach 1:
The patent replaces the mechanical/optical depth sensing system with a computational approach using machine learning models. The system uses a first ML model to estimate depth of a point of interest and a second ML model to estimate distances of pixels from that point, substituting physical depth sensors with algorithmic depth estimation from standard images.
Solution Approach 2:
The patent introduces machine learning models as intermediaries between the standard camera and the background replacement process. These models act as computational mediators that infer depth information without direct physical measurement, enabling accurate background replacement without additional hardware sensors.
2Measurement precision
If depth sensors are installed in user devices for AR applications, then depth measurement accuracy is improved, but manufacturing cost increases
Solution Approach 1:
The patent substitutes physical depth sensing hardware with software-based depth estimation using trained machine learning models. This eliminates the need for expensive depth sensors in each device, reducing manufacturing costs while maintaining depth measurement capability through computational methods.
Solution Approach 2:
The patent creates a computational copy of depth information through machine learning inference. Instead of physically measuring depth with sensors, the system generates depth maps by copying and adapting patterns learned from training data, providing accurate depth estimation without physical depth sensing hardware.
3Device complexity
If machine learning models are used for depth estimation, then system resource usage is reduced by eliminating depth sensors, but computational complexity increases
Solution Approach 1:
The patent divides the depth estimation task into two separate machine learning model stages: a first model estimates the depth of a point of interest, and a second model estimates distances of individual pixels from that point. This segmentation of the computational task improves efficiency and reduces the complexity of each individual model compared to a single comprehensive depth estimation system.
Data Source
AI summary
Methods and systems are disclosed for performing operations for applying augmented reality elements to a person depicted in an image. The operations include receiving an image that includes data representing a depiction of a person; extracting a portion of the image; applying a first machine learning model stage to the portion to predict a depth of a point of interest for the data representing the depiction of the person; applying a second machine learning model stage to the portion of the image to predict a relative depth of each pixel in the portion of the image to the predicted depth of the point of interest; generating dense depth reconstruction of the data representing the depiction of the person based on outputs of the first and second stages of the machine learning model; and applying one or more AR elements to the image based on the dense depth reconstruction.