Markerless Facial Motion Capture Using AI-Generated Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing facial motion capture systems are cumbersome, expensive, and require markers or additional sources like projected patterns, limiting their robustness, accuracy, and real-time capability.
Innovation Solution
A method using artificial intelligence engines to convert markerless facial images into digital representations by training with markered and markerless skin part images, employing generative adversarial networks to remove markers and create accurate digital mappings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If marker-based systems are used for facial motion capture, then measurement precision is improved, but device complexity and ease of operation deteriorate due to time-consuming marker application
Solution Approach 1:
The patent removes markers entirely from the facial motion capture system, extracting the tracking function from physical markers and implementing it through AI-based landmark detection on markerless images, thereby eliminating the time-consuming marker application process while maintaining measurement precision
Solution Approach 2:
The patent replaces the mechanical marker application process with an artificial intelligence-based image processing system that automatically detects facial landmarks in markerless images, substituting physical marker systems with computational algorithms to achieve both accuracy and operational efficiency
2Measurement precision
If multi-camera rigs or laser marker systems are used for three-dimensional motion capture, then measurement precision is improved, but device complexity and cost increase
Solution Approach 1:
The patent extracts the 3D motion capture capability from complex multi-camera and laser systems, achieving three-dimensional facial motion analysis through AI processing of standard 2D images, thereby eliminating the need for expensive and complicated hardware systems
Solution Approach 2:
The patent creates a digital 3D representation of facial motion by processing 2D images through artificial intelligence, generating accurate 3D landmark coordinates and motion data without requiring physical 3D scanning hardware, thus providing a simplified alternative to laser and multi-camera systems
3Ease of operation
If existing markerless systems are used, then ease of operation is improved, but reliability deteriorates due to requirements for additional sources like projected patterns or invisible paint
Solution Approach 1:
The patent removes all additional requirements such as projected patterns, invisible paint, and special lighting from the markerless system, achieving reliable facial motion capture using only standard digital images, thereby improving both ease of operation and system robustness
Solution Approach 2:
The patent enables the system to work with ordinary digital images that already exist, allowing the AI to extract facial motion data from standard photos or video frames without requiring any special preparation, additional equipment, or modified imaging conditions, thus achieving true markerless and robust operation
4Productivity
If real-time processing is implemented, then productivity is improved, but measurement precision may deteriorate due to processing speed requirements
Solution Approach 1:
The patent trains the artificial intelligence model in advance on large datasets of facial images and corresponding landmark annotations, performing the computationally intensive learning process beforehand so that during actual use, the pre-trained model can quickly and accurately process new images in real-time without sacrificing precision
Solution Approach 2:
The patent implements a dynamic processing approach where the AI model adapts its computation based on image complexity and required output speed, optimizing the balance between processing speed and accuracy by adjusting inference parameters and using efficient computational architectures suitable for real-time applications
Data Source
AI summary
A method of preparing a training data set for an artificial intelligence engine operable to convert markerless digital images of a target face into a mapping. The method comprises training an intermediate artificial intelligence by providing markered digital images of one or more skin parts to the intermediate artificial intelligence engine as a source training data set and providing markerless digital images of the one or more skin parts to the intermediate artificial intelligence engine as a target training data set. The intermediate artificial intelligence engine removes markers from markered digital images of the face of a first person to create corresponding synthetic markerless digital images of the face of the first person. The markered digital images of the face of the first person are converted into a mapping. The output training data set comprises the synthetic markerless digital images of the face of the first person and the mapping.