HMD Facial Tracking Patch Image Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Head-mounted display (HMD) devices obstruct the view of facial expressions and movements, making real-time communication less engaging and personal, as they cover a significant portion of the face, hindering the conveyance of emotions and intentions during virtual reality applications like video conferencing.
Innovation Solution
A method involving sensors and cameras that track facial movements and generate a patch image to simulate hidden facial features, merging this with a background image to create a composite view that appears as if the HMD device is removed, allowing real-time transmission of simulated facial expressions to other users.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If HMD device covers the face to provide virtual reality immersion, then virtual reality experience is improved, but facial expression visibility is lost
Solution Approach 1:
The patent creates a virtual copy of the user's face using 3D modeling and rendering techniques. Sensors track actual facial movements, and this data drives a digital twin of the user's face, which is then rendered and transmitted to other users. This copying approach preserves facial expression visibility while maintaining the physical HMD coverage for VR immersion.
Solution Approach 2:
The patent introduces an intermediary system consisting of sensors, processing units, and rendering software that mediates between the physical face (covered by HMD) and the visual representation needed for communication. This intermediary captures subtle facial movements through sensors and translates them into visible virtual facial expressions, bridging the gap between immersion and communication.
2Adaptability or versatility
If HMD device is worn to enable virtual reality applications, then immersion in virtual environment is improved, but real-time facial communication is hindered
Solution Approach 1:
The patent replaces the mechanical/physical system of direct facial visibility with an electronic sensing and rendering system. Instead of relying on optical capture of facial features, the system uses sensors to detect facial movements and electronically generates visual representations, substituting physical communication pathways with electronic ones.
Solution Approach 2:
The system enables users to communicate their facial expressions to others while wearing HMDs by making each user's device capture and transmit their own facial data. The processing and rendering occur locally or through coordinated systems, allowing users to serve their own communication needs without requiring external assistance or removing the HMD.
3Measurement precision
If sensors track facial movements under HMD, then facial expression simulation is improved, but device complexity increases
Solution Approach 1:
The patent divides the facial tracking system into multiple independent sensor modules, each responsible for detecting specific facial movements or features. This segmentation allows for specialized sensing approaches for different facial regions while distributing the processing load, making the complex system more manageable and potentially more accurate through focused measurement capabilities.
Data Source
AI summary
Various devices, arrangements and methods for managing communications using a head mounted display device are described. In one aspect, tracking data is generated at least in part by one or more sensors in a head mounted display (HMD) device. The tracking data indicates one or more facial movements of a user wearing the HMD device. A patch image is obtained based on the tracking data. The patch image is merged with a facial image. Various embodiments relate to the HMD device and other methods for generating and using the patch and facial images.