Self Presence in Artificial Reality Using Real-Time Image Capture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial reality systems fail to accurately display user representations, leading to disconnection and nausea due to inaccuracies in body tracking and reliance on computationally expensive procedures, resulting in lag and poor detail in computer-generated avatars.
Innovation Solution
The system captures real-time images of the user, applies a machine learning model to extract a self portion, and displays it as a self representation in the artificial reality environment, reducing computational load and improving accuracy and detail by using real-world images instead of computer-generated avatars.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If computer-generated avatars are used to represent the user in artificial reality, then the system can provide a visual representation of the user's body, but the computational cost increases and accuracy decreases leading to disconnection and nausea
Solution Approach 1:
The patent uses real-world camera images of the user's body parts as direct copies instead of generating synthetic computer-generated avatars. This approach captures actual visual data from the user's perspective and renders it in the artificial reality environment, providing accurate self-presence without the computational overhead of generating and rendering complex 3D avatar models in real-time
Solution Approach 2:
The patent replaces the mechanical/computational system of tracking body parts and mapping them to a kinematic model with a direct optical capture system. Instead of using sensors to track body position and mathematically constructing an avatar, the system uses cameras to directly capture images of the user's body parts from the user's perspective and renders these images directly, substituting computational modeling with optical capture
2Measurement precision
If computer-generated avatars with detailed body tracking are implemented, then self-presence accuracy improves, but processing time increases causing lag
Solution Approach 1:
The system performs preliminary capture of the user's body appearance and characteristics using cameras before the artificial reality session begins or during initialization. This pre-captured visual data is stored and then directly rendered during the session without requiring real-time computation of detailed body models, allowing for accurate self-presence representation with minimal processing delay during actual use
3Productivity
If real-time image capture and processing is performed to create self representations, then computational expense decreases, but image quality and detail may be reduced
Solution Approach 1:
The patent merges multiple camera feeds capturing different body parts (hands, feet, torso) into a single coherent self-representation in the artificial reality environment. By combining these real-world image captures and rendering them together with the artificial reality scene, the system achieves detailed and accurate self-presence representation while maintaining real-time processing efficiency through optimized image merging algorithms
Data Source
AI summary
The disclosed artificial reality system can provide a user self representation in an artificial reality environment based on a self portion from an image of the user. The artificial reality system can generate the self representation by applying a machine learning model to classify the self portion of the image. The machine learning model can be trained to identify self portions in images based on a set of training images, with portions tagged as either depicting a user from a self-perspective or not. The artificial reality system can display the self portion as a self representation in the artificial reality environment by positioning them in the artificial reality environment relative to the user's perspective in the artificial reality environment. The artificial reality system can also identify movements of the user and can adjust the self representation to match the user's movement, providing more accurate self representations.