Mirror-Based AR Experience with Skeletal Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual rendering systems face challenges in maintaining the illusion of virtual objects in real-world environments due to environmental conditions, user actions, and unanticipated visual interruptions.
Innovation Solution
The system dynamically adjusts the placement, positioning, and movement of virtual objects relative to a real-world object reference point, such as a person, by tracking skeletal joints and updating the virtual object's position in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If virtual objects are rendered in real-world environments, then engaging and entertaining augmented reality experiences are created, but environmental conditions and visual interruptions cause the virtual objects to disappear or behave erratically
Solution Approach 1:
The system dynamically adjusts the rendering of virtual objects based on real-time detection of environmental conditions and visual interruptions. The virtual object rendering is made adaptive by continuously monitoring camera feeds and adjusting placement, positioning, and movement to maintain the illusion of presence despite changing conditions.
Solution Approach 2:
The system implements feedback mechanisms by detecting visual interruptions and environmental changes, then using this information to adjust the virtual object rendering. The detection of skeletal joints and real-time updating of virtual object positions creates a closed-loop system that responds to environmental feedback to maintain reliability.
2Reliability
If the system tracks skeletal joints and updates virtual object position in real-time, then the illusion of virtual object presence is maintained, but computational complexity and processing requirements increase
Solution Approach 1:
The system uses a multi-functional approach where the camera system serves both as the primary imaging device and as a sensor for detecting skeletal joints and environmental conditions. The same hardware infrastructure is leveraged for multiple purposes: capturing video, tracking skeletal joints, detecting visual interruptions, and rendering virtual objects, thereby reducing overall system complexity.
Solution Approach 2:
The system performs self-service by using its own camera feeds and processing capabilities to detect skeletal joints and update virtual object positions without requiring external tracking equipment. The virtual rendering system monitors itself through its captured video streams, eliminating the need for separate complex tracking hardware.
Data Source
AI summary
Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving a video that depicts a person. The operations further include identifying a set of skeletal joints of the person. The operations further include identifying a pose of the person depicted in the video based on positioning of the set of skeletal joints (or detecting a hand pose, detecting a mirror frame, or detecting a mobile device). The operations further include determining, based on the pose of the person (or detecting a hand pose, detecting a mirror frame, or detecting a mobile device), that the video comprises a mirror reflection of the person. The operations further include, in response to determining that the video comprises the mirror reflection of the person, causing display of a 3D virtual object in the video.