Facial Image Reenactment with Separate Identity and Expression Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing facial image analysis techniques fail to distinguish between appearance and emotional characteristics, leading to misclassification and deterioration in performance, particularly in few-shot settings where the identity of the target face does not coincide with the driver face.
Innovation Solution
A method and apparatus that extract landmarks from both driver and target images, generate feature maps based on pose and expression information, and combine these maps to create a reenacted image with the identity of the target face and the pose and facial expression of the driver face, using a landmark transformer, encoders, and a decoder.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing facial image analysis techniques are used to process facial landmarks, then the processing can be performed, but the appearance characteristics and emotional characteristics cannot be distinguished, leading to misclassification and performance deterioration
Solution Approach 1:
The patent segments facial characteristics into two distinct types: appearance characteristics (identity, facial structure) and emotional characteristics (expressions, poses). This is achieved by separately processing and representing these characteristics through different feature extraction mechanisms, allowing them to be distinguished and processed independently, thereby resolving the contradiction between processing capability and characteristic distinction.
2Productivity
If facial landmarks are processed without distinguishing appearance and emotional characteristics, then the processing is simpler, but misclassification occurs and performance deteriorates in few-shot settings
Solution Approach 1:
The system segments the facial analysis task into separate modules for appearance characteristics and emotional characteristics. This segmentation allows each module to specialize in its specific aspect, improving reliability for emotion classification in few-shot settings while maintaining processing efficiency through modular architecture.
Solution Approach 2:
The patent introduces an intermediary mechanism that processes appearance and emotional characteristics separately before integrating them. This intermediary processing layer ensures that appearance features (which may cause misclassification) are handled independently from emotional features, thereby improving classification accuracy without significantly reducing processing efficiency.
3Adaptability or versatility
If the target face identity does not coincide with the driver face, then the reenactment can be performed, but the performance deteriorates in few-shot settings
Solution Approach 1:
The patent segments the reenactment process into separate processing of appearance characteristics and emotional characteristics. By decoupling these aspects, the system can apply emotional expressions from the driver face to the target face identity without the interference of identity mismatch, thereby maintaining high reenactment quality even when driver and target identities differ.
Solution Approach 2:
The system applies local quality by treating different facial regions and characteristics differently. Appearance characteristics (identity-related) and emotional characteristics (expression-related) are processed with different weights and methods, allowing the system to preserve target face identity while applying driver face expressions, thus maintaining high quality in cross-identity reenactment scenarios.
Data Source
AI summary
A method of generating a reenacted image includes: extracting a landmark from each of a driver image and a target image; generating a driver feature map based on pose information and expression information of a first face shown in the driver image; generating a target feature map and a pose-normalized target feature map based on style information of a second face shown in the target image; generating a mixed feature map by using the driver feature map and the target feature map; and generating the reenacted image by using the mixed feature map and the pose-normalized target feature map.