HMD Camera Placement for Markerless Facial Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional virtual reality systems provide limited graphical representations of users, lacking immersive facial and body movements due to the use of dedicated peripherals and markers, which are not suitable for portable, lightweight, high-performance headsets.
Innovation Solution
A virtual reality or augmented reality head-mounted display equipped with multiple image capture devices positioned to capture various portions of the user's face, including those outside and within the headset, coupled with a controller that analyzes images to generate realistic facial and body animations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If dedicated peripherals and markers are used for facial tracking, then tracking accuracy is improved, but device complexity and user burden increase
Solution Approach 1:
The patent extracts the tracking markers from the system entirely, replacing them with natural facial features as tracking points. The system now uses only the headset-mounted cameras to capture and analyze facial movements without requiring any external markers or peripherals, thus reducing device complexity while maintaining tracking functionality.
Solution Approach 2:
The patent creates a virtual copy of the user's facial features by capturing images with headset cameras and generating 3D models that replicate facial geometry and expressions. This digital twin approach allows accurate tracking without physical markers, resolving the contradiction between tracking precision and system simplicity.
2Measurement precision
If multiple image capture devices are positioned within the HMD to capture enclosed facial portions, then facial tracking coverage is improved, but device weight and complexity increase
Solution Approach 1:
The patent makes each image capture device multi-functional by positioning them to serve multiple tracking purposes simultaneously. The cameras capture both external facial features (mouth, chin) and enclosed features (eyes, nose) depending on head orientation, eliminating the need for separate dedicated cameras for each facial region and reducing overall device weight.
Solution Approach 2:
The patent implements dynamic tracking where the system adapts which cameras are active based on real-time head orientation and facial visibility. This allows the headset to use fewer cameras at any given moment while still maintaining comprehensive facial coverage, reducing the number of required image capture devices and thereby reducing weight.
3Device complexity
If conventional avatar systems with predetermined movements are used, then system simplicity is maintained, but immersion and realism are reduced
Solution Approach 1:
The patent implements real-time feedback loops where captured facial images are continuously analyzed to extract movement data, which is then immediately applied to update the virtual avatar's expressions and movements. This closed-loop system creates realistic, responsive avatars that mirror the user's actual facial movements, significantly improving immersion while using computationally efficient algorithms.
Solution Approach 2:
The patent replaces the mechanical approach of predetermined animation sequences with a data-driven system that uses image processing and machine learning to generate avatar movements. This substitution allows the system to maintain simplicity through automated processing while achieving high realism through accurate facial movement capture and replication.
Data Source
AI summary
A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes multiple image capture devices positioned within and on the HMD to capture portions of a face of a user wearing the HMD. Multiple image capture devices are included within the HMD to capture different portions of the face of the user within the HMD, and one or more other image capture devices are positioned to capture portions of the face of the user external to the HMD. Captured images from various image capture devices may be communicated to a console or a controller that generates a graphical representation of the user's face based on the captured images.