Pixel Depth Estimation for AR Object Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality systems in messaging applications face challenges in accurately recognizing and modifying entire objects, such as a user's body, without depth sensors, leading to poor image quality and failure in applying visual effects, especially when objects are at a distance or multiple objects are present in the image.
Innovation Solution
The system employs machine learning models to crop and analyze portions of images, estimate depth, and generate dense depth reconstructions, allowing for realistic application of AR effects on objects without affecting the background, thereby improving image quality and effect application.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth sensors are used to accurately recognize and modify objects in AR, then measurement precision and reliability improve, but device complexity and manufacturing cost increase
Solution Approach 1:
The patent replaces the mechanical/optical depth sensing system with a computational approach using machine learning models. The system uses 2D image data from standard cameras combined with ML-based depth estimation algorithms to achieve depth measurement without physical depth sensors, thereby reducing device complexity while maintaining measurement precision.
Solution Approach 2:
The patent introduces machine learning models as an intermediary between the camera and the AR rendering system. These models process 2D image data and generate depth maps, serving as a computational mediator that enables depth-based AR effects without requiring direct hardware depth sensing capability.
2Measurement precision
If the entire image is processed to recognize objects, then measurement precision improves, but use of energy and processing time increase
Solution Approach 1:
The patent applies segmentation by dividing the image processing task into regions of interest (objects to be augmented) and background regions. The machine learning model identifies and processes only the relevant object portions, reducing the overall processing energy while maintaining recognition precision through focused analysis of key areas.
Solution Approach 2:
The patent uses partial action by processing only the necessary portions of the image (objects of interest) rather than the entire image. The system applies depth estimation and AR effects selectively to identified objects, reducing computational energy consumption while achieving sufficient measurement precision for the intended application.
3Productivity
If AR effects are applied to entire images, then productivity improves, but manufacturing precision and object-specific accuracy worsen
Solution Approach 1:
The patent applies local quality by differentiating processing between objects and background. AR effects are applied with appropriate quality and depth accuracy only to identified objects, while background regions receive different or no processing. This enables object-specific accuracy without requiring uniform high-quality processing of the entire image.
Data Source
AI summary
Methods and systems are disclosed for performing operations for applying augmented reality elements to a person depicted in an image. The operations include receiving an image that includes data representing a depiction of a person; extracting a portion of the image; applying a first machine learning model stage to the portion to predict a depth of a point of interest for the data representing the depiction of the person; applying a second machine learning model stage to the portion of the image to predict a relative depth of each pixel in the portion of the image to the predicted depth of the point of interest; generating dense depth reconstruction of the data representing the depiction of the person based on outputs of the first and second stages of the machine learning model; and applying one or more AR elements to the image based on the dense depth reconstruction.