Image Warping of Real-World Objects Without Depth Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AR systems require depth sensors for image modification, increasing device cost and complexity, and struggle to accurately replace backgrounds or apply visual effects to whole bodies, especially when users are at a distance or multiple users are present, and they fail to account for physical properties like density and weight, leading to inefficient processing and unrealistic modifications.
Innovation Solution
A machine learning model estimates a warping field and segmentation mask to deform real-world objects in images without depth sensors, allowing for efficient and realistic application of AR graphics, considering movement and pose information, and reducing processing complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth sensors are used for image modification, then image modification accuracy is improved, but device cost and complexity increase
Solution Approach 1:
The patent extracts the depth sensing capability from a dedicated depth sensor hardware component and implements it through software-based monocular depth estimation using machine learning models. This removes the need for physical depth sensors while maintaining depth information extraction capability, thereby reducing device complexity while preserving measurement precision for image modification
Solution Approach 2:
The patent replaces the mechanical/optical depth sensing system with a computational approach using neural networks that process monocular images to estimate depth maps. This substitution eliminates complex hardware while achieving comparable or superior depth estimation accuracy through algorithms that infer three-dimensional structure from two-dimensional images
2Productivity
If traditional background replacement methods are used, then processing speed is maintained, but accuracy deteriorates when users are at a distance or multiple users are present
Solution Approach 1:
The patent segments the image processing task into multiple components: monocular depth estimation to separate foreground users from background, semantic segmentation to identify user regions, and pose estimation to track body parts. This multi-level segmentation enables accurate background replacement even for distant users or multiple users by processing different aspects of the image independently and combining results
Solution Approach 2:
The patent introduces depth maps and segmentation masks as intermediary data structures that bridge the gap between input images and final background replacement. These intermediaries encode spatial and semantic information that guide precise user isolation and background substitution, maintaining accuracy across varying user distances and configurations
3Productivity
If visual effects are applied without considering physical properties, then processing efficiency is improved, but realism deteriorates
Solution Approach 1:
The patent applies dynamic pose estimation that tracks user body positions and orientations across video frames, enabling visual effects to adapt to changing user poses in real-time. This dynamic approach maintains processing efficiency through optimized temporal processing while significantly improving realism by ensuring visual effects correctly follow user movements and maintain proper spatial relationships
Data Source
AI summary
Methods and systems are disclosed for performing real-time deforming operations. The system receives an image that includes a depiction of a real-world object. The system applies a machine learning model to the image to generate a warping field and segmentation mask, the machine learning model trained to establish a relationship between a plurality of training images depicting real-world objects and corresponding ground-truth warping fields and segmentation masks associated with a target shape. The system applies the generated warping field and segmentation mask to the image to warp the real-world object depicted in the image to the target shape.