Real-Time Facial Performance Capture Using Segmentation Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing facial capture and recognition systems struggle with real-time processing and accuracy, especially when faced with occlusions such as hair, hands, or shadows, requiring additional equipment like depth sensors and being unsuitable for general public use with standard devices.
Innovation Solution
A system utilizing well-trained convolutional neural networks, specifically the VGG-16 network, for real-time facial segmentation and performance capture from RGB input, which effectively handles occlusions by generating probability maps and binary masks, enabling accurate facial tracking and replacement without the need for depth sensors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional facial capture systems use fiducial markers and extensive post-processing, then measurement precision is improved, but productivity deteriorates due to hours of processing time required
Solution Approach 1:
The patent replaces traditional mechanical marker-based capture systems with a deep learning-based neural network that directly processes RGB images to extract facial performance data in real-time, eliminating the need for physical markers and extensive post-processing
Solution Approach 2:
The system changes the processing approach from traditional image processing parameters to deep learning parameters, using convolutional neural networks to directly map pixel data to facial performance parameters, enabling real-time processing
2Ease of operation
If facial recognition systems rely on natural landmark detection, then ease of operation is improved, but reliability deteriorates when facial landmarks are occluded
Solution Approach 1:
The patent transitions from two-dimensional landmark detection to three-dimensional facial performance modeling, enabling the system to infer occluded facial regions by projecting 2D image data onto a 3D facial model, thereby maintaining reliability under occlusion
Solution Approach 2:
The system introduces a three-dimensional facial model as an intermediary between the two-dimensional occluded image and the required facial performance data, allowing the neural network to infer complete facial geometry even when portions are blocked
3Reliability
If systems add depth sensors to handle occlusions, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent creates a virtual copy of the three-dimensional facial model from two-dimensional RGB images using deep learning, allowing the system to handle occlusions through computational modeling rather than requiring physical depth sensors
Solution Approach 2:
The system replaces mechanical depth sensors with a software-based deep learning solution that infers three-dimensional facial geometry from two-dimensional images, eliminating the need for additional hardware while maintaining occlusion handling capability
4Productivity
If convolutional neural networks are trained with facial data for real-time operation, then productivity is improved, but measurement precision deteriorates when dealing with occlusions
Solution Approach 1:
The patent enhances two-dimensional neural network processing by integrating three-dimensional facial model constraints, allowing the system to maintain real-time performance while improving accuracy through 3D geometric consistency checks
Data Source
AI summary
There is disclosed a system and method of performing facial recognition from RGB image data. The method includes generating a lower-resolution image from the RGB image data, performing a convolution of the lower-resolution image data to derive a probability map identifying probable facial regions and a probable non-facial regions, and performing a first deconvolution on the lower-resolution image using a bilinear interpolation layer to derive a set of coarse facial segments. The method further includes performing a second deconvolution on the lower-resolution image using a series of unpooling, deconvolution, and rectification layers to derive a set of fine facial segments, concatenating the set of coarse facial segments to the set of fine facial segments to create an image matrix made up of a set of facial segments, and generating a binary facial mask identifying probable facial regions and probable non-facial regions from the image matrix.