Machine Learning Face Reconstruction for Headset-Occluded Regions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer simulations, such as VR or AR games, often occlude players' faces due to headsets, hindering social interaction and enjoyment by obscuring full facial views among teammates and opponents.
Innovation Solution
A machine learning model, utilizing RNNs like LSTM, reconstructs full-face images by training on pairs of images with and without occluding headsets, and optionally overlays transparent headsets to maintain realism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If players wear headsets during computer simulations, then immersion and gameplay experience are improved, but facial visibility and social interaction are worsened
Solution Approach 1:
The system creates a virtual copy of the player's face using machine learning models. The ML model generates a synthetic full-face image that replicates the player's facial features, expressions, and characteristics. This virtual face copy is then displayed to other players, allowing them to see the player's face even though the physical headset occludes it in the real world.
Solution Approach 2:
The patent introduces an intermediary system consisting of cameras, ML models, and display interfaces. Cameras capture partial facial data around the headset, the ML model processes this data to generate the complete virtual face, and the display interface presents this reconstructed face to other players. This intermediary chain bridges the gap between the occluded real face and the visible virtual representation.
2Reliability
If headsets occlude the face, then simulation functionality is improved, but social interaction and player enjoyment are worsened
Solution Approach 1:
The system applies transparency and opacity adjustments to the virtual headset overlay. The virtual headset is rendered with partial transparency, allowing viewers to see through it to the reconstructed face beneath. This transparency effect maintains the visual cue that the player is wearing a headset while simultaneously allowing facial features to remain visible, thus preserving both simulation authenticity and social interaction.
3Ease of operation
If full-face images are reconstructed, then social interaction is improved, but computational complexity and processing requirements are worsened
Solution Approach 1:
The system processes only the necessary portions of facial data rather than attempting to reconstruct the entire face from complete information. The ML model focuses on generating key facial features and expressions from the limited visible areas around the headset, using predictive algorithms to infer the occluded regions. This partial processing approach reduces computational burden while maintaining sufficient social interaction quality.
Data Source
AI summary
An image of a computer game player wearing a headset that occludes part of the face is input to a trained machine learning (ML) model, which outputs in response a full-face image that is not occluded for use in, e.g., social network settings related to the game.