HMD Facial Tracking via Integrated Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional virtual reality systems provide limited graphical representations of users, lacking immersive facial expressions due to the need for dedicated peripherals and markers, which are unsuitable for portable, lightweight, and high-performance headsets.
Innovation Solution
A head-mounted display with integrated facial sensors and a controller that uses machine learning to capture and process facial expressions, projecting a realistic facial animation model onto the virtual environment without the need for external markers or peripherals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional facial tracking systems use dedicated peripherals and markers, then facial expression tracking accuracy is improved, but device complexity and weight increase
Solution Approach 1:
The patent combines facial tracking sensors directly into the HMD device itself, merging the tracking function with the display device. This eliminates the need for separate dedicated peripheral devices and markers, reducing overall system complexity while maintaining tracking capability through integrated sensors that capture facial expressions directly.
Solution Approach 2:
The patent extracts the facial tracking functionality from separate peripheral devices and integrates it into the HMD. By taking out the tracking sensors and embedding them within the headset structure, the system eliminates external markers and dedicated tracking peripherals, simplifying the overall system architecture.
2Measurement precision
If conventional facial tracking systems use dedicated peripherals and markers, then facial expression tracking accuracy is improved, but portability and comfort deteriorate
Solution Approach 1:
By merging facial tracking sensors into the HMD structure, the patent eliminates the need for separate peripheral devices and external markers. This integration reduces the total weight that would otherwise be required for dedicated tracking equipment, improving portability and user comfort while maintaining tracking accuracy.
3Measurement precision
If conventional facial tracking systems use markers and dedicated peripherals, then facial expression tracking is achieved, but user immersion in virtual environment deteriorates
Solution Approach 1:
The patent extracts and eliminates the need for external markers and dedicated peripheral devices by integrating sensors directly into the HMD. This removal of extraneous tracking components allows users to interact naturally with the virtual environment without being separated by physical markers or additional equipment, thereby enhancing immersion.
4Device complexity
If limited facial expressions are used in avatars, then device complexity is reduced, but user experience immersion deteriorates
Solution Approach 1:
The patent enables the system to automatically capture and process a full range of facial expressions through integrated sensors and machine learning algorithms. The system serves itself by autonomously tracking multiple facial muscle movements and translating them into corresponding avatar expressions, eliminating the need for pre-programmed limited expressions while maintaining system efficiency.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables a more immersive user experience by accurately reflecting facial expressions, enhancing the realism of graphical representations within virtual reality environments without the limitations of traditional tracking systems.
Implementation Method 1
a facial sensor is a camera or other image capture device positioned to capture images of portions of the user's face
Implementation Method 2
the facial sensor also includes an illumination device configured to illuminate the portions of the user's face captured by the facial sensor
Data Source
AI summary
A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes various facial sensors, such as cameras, that capture images of portions of the user's face outside of the HMD. For example, multiple facial sensors capture images of a portion of the user's face below the HMD. Through image analysis, points of the portion of the user's face are identified from the images and their movement is tracked. The identified points are mapped to a three dimensional model of a face. Additionally, a parametric representation of the user's face is determined for each captured image, resulting in various representations indicating the user's facial expressions. From the parametric representations and transforms mapping the captured images to three dimensions, a rendering model is used and applied to the three dimensional model of the face to render the user's facial expressions.