Body Normal Network Rendering for Depth-Free Clothing AR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality systems require depth sensors to modify images, increasing device cost and complexity, and struggle to recognize and apply visual effects to a user's whole body, especially when users are at a distance or multiple users are present, leading to poor image quality and ineffective background replacement.
Innovation Solution
A system that segments articles of clothing using machine learning techniques without depth sensors, allowing for the application of visual effects like lighting changes and material modifications based on pixel angles relative to the camera, enabling realistic AR experiences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If depth sensors are used to modify images in AR systems, then image modification capability is improved, but device cost and complexity increase
Solution Approach 1:
The patent extracts the depth sensing function from dedicated hardware sensors and implements it through software-based monocular depth estimation using standard RGB cameras. The system removes the requirement for depth sensors by extracting depth information algorithmically from single-viewpoint images through neural network processing.
Solution Approach 2:
The patent replaces the mechanical/optical depth sensing system with a computational approach using machine learning models. Instead of using physical depth sensors to capture depth data, the system uses software-based monocular depth estimation algorithms that process standard RGB images to infer depth information.
2Measurement precision
If depth sensors are required for AR image modification, then measurement precision is improved, but ease of manufacture worsens
Solution Approach 1:
The patent creates a computational copy of depth information from standard RGB images through monocular depth estimation. Instead of requiring physical depth sensors to capture actual depth data, the system generates a simulated depth map that replicates depth information algorithmically from visual input alone.
Solution Approach 2:
The patent changes the fundamental parameter for depth acquisition from hardware-based optical sensing to software-based computational estimation. The system transforms the approach from directly measuring depth through sensors to inferring depth parameters through machine learning analysis of RGB image data.
3Adaptability or versatility
If the system needs to recognize whole body for AR effects, then adaptability is improved, but difficulty of detecting and measuring increases
Solution Approach 1:
The patent segments the image processing task into specialized components: pose estimation networks detect body keypoints and skeletal structure, while monocular depth estimation networks generate depth maps. This segmentation allows each component to specialize in specific aspects of body recognition, improving overall accuracy and adaptability.
Solution Approach 2:
The patent adds depth as an additional dimension to the traditional 2D pose estimation problem. By incorporating monocular depth estimation, the system transforms planar image data into three-dimensional spatial understanding, enabling more accurate whole-body recognition and AR effect application.
Data Source
AI summary
Methods and systems are disclosed for performing operations for applying augmented reality elements to a fashion item. The operations include receiving an image that includes a depiction of a person wearing a fashion item. The operations include generating a segmentation of the fashion item worn by the person depicted in the image. The operations include extracting a portion of the image corresponding to the segmentation of the fashion item; estimating an angle of each pixel in the portion of the image relative to a camera used to capture the image. The operations include applying one or more augmented reality elements to the fashion item in the image based on the estimated angle of each pixel in the portion of the image relative to the camera used to capture the image.