AR Motion Transfer via Skeletal Tracking and ML Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Virtual objects in augmented reality experiences often disappear or behave erratically due to environmental conditions and unanticipated visual interruptions, breaking the illusion of their presence in real-world environments.
Innovation Solution
The augmentation system dynamically adjusts the placement and movement of virtual objects relative to real-world objects by tracking skeletal joints and using machine learning to predict movement, ensuring seamless integration and consistent rendering in three-dimensional spaces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If virtual objects are rendered in real-world environments using augmented reality systems, then engaging and entertaining experiences are created, but environmental conditions and visual interruptions cause the virtual objects to disappear or behave erratically, breaking the illusion of presence
Solution Approach 1:
The system performs preliminary actions by capturing images and video data of the real-world environment before rendering virtual objects. This pre-captured data serves as a stable reference that compensates for subsequent environmental changes and visual interruptions, allowing the virtual objects to maintain consistent placement and behavior even when real-time camera views are disrupted.
Solution Approach 2:
The system introduces an intermediary mechanism by using captured environmental data as a mediator between the camera and the virtual object rendering. This intermediary reference data allows the system to reconstruct accurate spatial relationships and object placements even when direct visual input is interrupted or degraded by environmental conditions.
2Measurement precision
If the system captures and processes images and video data to determine real-world object positions, then accurate virtual object placement is achieved, but computational complexity and processing time increase
Solution Approach 1:
The system performs image and video capture in advance before the actual virtual object rendering takes place. This preliminary data collection allows complex processing to be done beforehand, reducing real-time computational requirements while maintaining high measurement precision for object positioning.
Solution Approach 2:
The system uses the device's own camera and processing capabilities to capture and analyze environmental data, eliminating the need for external sensors or complex additional hardware. The device serves itself by utilizing its built-in resources for both data capture and processing, simplifying the overall system architecture.
Data Source
AI summary
Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving a video that depicts a person; identifying a set of skeletal joints corresponding to limbs of the person; tracking 3D movement of the set of skeletal joints corresponding to the limbs of the person in the video; causing display of a 3D virtual object that has a plurality of limbs including one or more extra limbs than the limbs of the person in the video; and moving the one or more extra limbs of the 3D virtual object based on the movement of the set of skeletal joints corresponding to the limbs of the person in the video.