Frontalizing Pose-Variant Faces via 3DMM-GAN for Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current face recognition systems face significant accuracy drops when dealing with large pose variant faces, requiring expensive labeled data and carefully designed feature learning algorithms, which is inefficient and impractical for real-world applications.
Innovation Solution
A video retrieval system utilizing a three-dimensional Morphable Model (3DMM) conditioned Generative Adversarial Network (GAN) to estimate 3DMM coefficients for input images, generating synthetic frontal face images and determining identity, which expands pose range to include extreme profile views without the need for extensive training data or identity-preserving encoders.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If general face recognition engines are used, then recognition speed is maintained, but accuracy drops significantly for large pose variant faces
Solution Approach 1:
The system performs preliminary face alignment and frontalization before recognition by estimating 3DMM coefficients and generating a frontalized face image from the input pose-variant face. This preprocessing step transforms the pose-variant face into a frontal view, enabling accurate recognition without requiring the recognition engine to handle pose variations directly.
Solution Approach 2:
The system introduces 3DMM coefficients and a generator network as intermediaries between the input pose-variant face and the recognition engine. The generator acts as a mediator that translates pose-variant faces into frontalized representations, while 3DMM coefficients serve as an intermediate representation capturing geometric information for accurate frontalization.
2Measurement precision
If pose-invariant feature learning is implemented, then recognition accuracy for large pose variants improves, but expensive labeled large scale data and carefully designed feature learning algorithms are required
Solution Approach 1:
The system replaces complex pose-invariant feature learning algorithms with a simpler generator-based approach. Instead of designing sophisticated feature learning architectures to handle pose variations, the system substitutes this with a frontalization generator that transforms pose-variant faces into frontal views, allowing standard recognition engines to work effectively.
Solution Approach 2:
The system changes the parameter representation by estimating 3DMM coefficients (shape, expression, pose parameters) from the input face and using these parameters to generate frontalized images. This parameter-based approach simplifies the problem from learning pose-invariant features directly to manipulating explicit geometric parameters for frontalization.
3Productivity
If the face area in the image is small, then processing speed is maintained, but recognition accuracy under large pose variations deteriorates
Solution Approach 1:
The system performs preliminary frontalization and face alignment before recognition processing. By estimating 3DMM coefficients and generating a frontalized face image in advance, the system ensures that the recognition engine receives a properly aligned and enlarged facial representation, improving accuracy without significantly impacting overall processing speed.
Data Source
AI summary
A video retrieval system is provided that includes a server for retrieving video sequences from a remote database responsive to a text specifying a face recognition result as an identity of a subject of an input image. The face recognition result is determined by a processor of the server, which estimates, using a 3DMM conditioned Generative Adversarial Network, 3DMM coefficients for the subject of the input image. The subject varies from an ideal front pose. The processor produces a synthetic frontal face image of the subject of the input image based on the input image and coefficients. An area spanning the frontal face of the subject is made larger in the synthetic than in the input image. The processor provides a decision of whether the synthetic image subject is an actual person and provides the identity of the subject in the input image based on the synthetic and input images.