AR Foot Tracking via Neural Network Decoders
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality systems fail to accurately track the placement, position, and orientation of users' feet in images or videos, leading to unrealistic placements of virtual shoes and increased processing complexity due to reliance on complex algorithms like Perspective-N-Point (PNP), which consumes significant resources and introduces errors.
Innovation Solution
The implementation of a machine learning technique that uses a neural network to detect and track feet in real-time, eliminating the need for PNP by providing dense outputs for each pixel, allowing for accurate 3D rotation, translation, and scale estimation of feet, and selecting appropriate virtual shoe objects based on foot classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex algorithms like Perspective-N-Point (PNP) are used to track feet, then foot tracking can be achieved, but processing complexity and resource consumption increase significantly
Solution Approach 1:
The patent replaces complex geometric algorithms (PNP) with a machine learning-based neural network system. The neural network directly processes image data to output foot center coordinates, orientation angles, and depth information, eliminating the need for complex iterative geometric calculations while achieving comparable or superior tracking accuracy.
Solution Approach 2:
The system transforms the output parameters from traditional PNP methods (which require multiple iterative calculations) into direct neural network outputs including 2D center coordinates, 3D orientation angles (roll, pitch, yaw), and depth values. This parameter transformation enables more efficient processing while maintaining tracking precision.
2Measurement precision
If complex algorithms like Perspective-N-Point (PNP) are used to track feet, then foot tracking can be achieved, but system resource consumption increases
Solution Approach 1:
The patent replaces computationally intensive geometric algorithms with a neural network model that processes images through learned feature representations. This substitution significantly reduces processing time and resource consumption while maintaining or improving foot tracking accuracy through data-driven feature extraction.
Solution Approach 2:
The system uses pre-trained neural network models that have learned optimal feature representations from training data. By copying the learned patterns rather than performing complex real-time calculations, the system achieves efficient inference with reduced computational resource requirements.
3Productivity
If traditional algorithms are used for foot tracking, then processing can be performed, but errors are introduced and accuracy decreases
Solution Approach 1:
The neural network system incorporates feedback mechanisms through training on labeled data, where the model continuously refines its predictions based on ground truth annotations. This feedback loop enables the system to correct errors and improve accuracy over time, achieving superior performance compared to static algorithms.
Solution Approach 2:
The patent replaces traditional algorithms that are prone to geometric errors and assumptions with a neural network that directly learns accurate foot detection and tracking from image data, eliminating systematic errors inherent in algorithmic approaches while maintaining processing efficiency.
Data Source
AI summary
Aspects of the present disclosure involve a system for presenting AR items. The system performs operations including: receiving an image that includes a depiction of a first real-world body part in a real-world environment; applying a machine learning technique to the image to generate a plurality of dense outputs each associated with a respective pixel of a plurality of pixels in the image; applying a first task-specific decoder to the plurality of dense outputs to identify a pixel corresponding to a center of the first real-world body part; applying a second task-specific decoder using the identified pixel to retrieve a 3D rotation, translation and scale of first real-world body part from the plurality of dense outputs; modifying an AR object based on the 3D rotation, translation, and scale of first real-world body part; and modifying the image to include a depiction of the modified AR object.