Outward-Facing Display for Real-Time Face Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic devices, such as head-mounted displays (HMDs) and wearable devices, obstruct the view of a user's face, making it difficult for others to see the user's eyes and facial expressions.
Innovation Solution
The implementation of systems and methods that display a view of a device user's face portion on an outward-facing display, using a combination of live eye camera data and guest data such as interpupillary distance (IPD) and skin tone, to create a realistic and 3D spatially accurate representation of the user's face.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a head-mounted display or wearable device is positioned in front of the user's face, then the user can access the device functionality, but others cannot see the user's eyes and facial expressions
Solution Approach 1:
The system creates a virtual copy of the user's face portion using sensor data, eye tracking information, and 3D modeling. This digital replica is then displayed on the outward-facing display, allowing observers to see what the user cannot physically show while wearing the device. The copy includes dynamic elements like eye movements and facial expressions captured by sensors.
Solution Approach 2:
The outward-facing display acts as an intermediary between the user's hidden face and external observers. It mediates the communication by presenting synthesized visual information that bridges the physical obstruction caused by the device, enabling social interaction without removing the wearable equipment.
2Speed
If live sensor data is used to capture the user's face, then real-time representation is achieved, but data completeness is insufficient for guest users without face enrollment
Solution Approach 1:
The system merges multiple data sources including live sensor data from eye cameras, depth sensors, and hand enrollment information to create a complete facial representation. For guest users, it combines available real-time data with stored biometric information and 3D mesh models to compensate for missing enrollment data, achieving both speed and completeness.
Solution Approach 2:
The system performs preliminary actions by using hand enrollment data and device settings (such as interpupillary distance) in advance to prepare baseline facial parameters. This preliminary preparation enables rapid generation of face representations for guest users without requiring time-consuming face scanning procedures.
3Loss of time
If a 3D mesh model is generated from limited data, then face representation is achieved quickly, but accuracy may be reduced
Solution Approach 1:
The system dynamically adjusts parameters such as interpupillary distance, skin tone, and facial geometry based on available data sources. For guest users, it uses device settings and sensor measurements to estimate and refine these parameters in real-time, maintaining acceptable accuracy while enabling rapid representation generation without full enrollment.
Data Source
AI summary
Various implementations disclosed herein include devices, systems, and methods that present a view of a device user's face portion, that would otherwise be blocked by an electronic device positioned in front of the face, on an outward-facing display of the user's device. The view of the user's face portion may be configured to enable observers to see the user's eyes and facial expressions as if they were seeing through a clear device at the user's actual eyes and facial expressions. Various techniques are used to provide views of the user's face that are realistic, that show the user's current facial appearance, and/or that present the face portion with 3D spatial accuracy, e.g., each eye appearing to be in its actual 3D position.