3D Bitmoji Facial Tracking for Real-Time AR Expression Overlay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality systems lack the ability to dynamically generate and overlay realistic 3D representations of users' facial expressions in real-time, limiting the interactivity and engagement of augmented reality interfaces.
Innovation Solution
An augmented expression system that tracks facial landmarks in real-time to generate and animate a 3D bitmoji, allowing users to interact with and share composite presentations that include their animated 3D representation, using facial tracking and modeling to mimic expressions and enable dynamic overlays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing augmented reality systems are used, then basic AR display functionality is provided, but the ability to dynamically generate and overlay realistic 3D representations of users' facial expressions in real-time is lacking
Solution Approach 1:
The system creates a 3D digital twin (bitmoji) that copies and mimics the user's facial expressions and movements in real-time. This allows realistic 3D representations to be generated without requiring complex scanning equipment, as the system captures and replicates facial dynamics through software-based modeling and tracking.
Solution Approach 2:
The system transitions from 2D camera input to 3D augmented reality output by generating three-dimensional bitmoji models that overlay virtual elements onto the real-world view. This dimensional transformation enables realistic 3D facial representations while maintaining relatively simple device requirements.
2Productivity
If real-time facial tracking is implemented, then interactive augmented reality displays are achieved, but processing requirements and computational load increase
Solution Approach 1:
The system segments facial tracking into discrete landmark points that can be independently detected and tracked. This segmentation approach reduces computational complexity by breaking down the complex task of facial analysis into manageable components, enabling real-time processing with lower energy consumption.
Solution Approach 2:
The system focuses on tracking specific key facial landmarks rather than analyzing every pixel or detail of the face. This partial action approach provides sufficient data for realistic 3D representation while significantly reducing the computational load and energy requirements compared to full-face analysis.
3Ease of operation
If 3D bitmoji overlay is added to existing AR systems, then user engagement is enhanced, but the device complexity and development requirements increase
Solution Approach 1:
The system integrates multiple functions into a unified framework: facial landmark detection, 3D model generation, real-time animation, and AR overlay are combined in a single system. This multi-functionality approach enhances user engagement while managing complexity through integrated design rather than separate components.
Solution Approach 2:
The system uses an intermediary 3D bitmoji model that mediates between the user's real facial expressions and the augmented reality display. This intermediary representation simplifies the integration process by providing a standardized intermediate format that can be easily overlaid onto various AR contexts without requiring direct complex mappings.
Data Source
AI summary
Embodiments described herein relate to an augmented expression system to generate and cause display of a specially configured interface to present an augmented reality perspective. The augmented expression system receives image and video data of a user and tracks facial landmarks of the user based on the image and video data, in real-time to generate and present a 3-dimensional (3D) bitmoji of the user.