Talking Face Generation With Canonical Motion Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating talking faces struggle to control detailed facial attributes such as head posture, eyebrow movement, and eye movement, and require additional supervision or facial keypoints, limiting their application in real-world scenarios.
Innovation Solution
A face image generation method using a multimodal latent code fusion process, involving a visual encoder, audio encoder, canonical encoder, and multimodal motion encoder, to generate a talking face image without additional supervision, ensuring orthogonality between identity and motion features through a generative adversarial network (GAN).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If deep learning-based talking face synthesis uses only RGB format supervision, then the implementation is simple, but the control of head posture and detailed facial attributes is insufficient
Solution Approach 1:
The patent introduces a canonical space as an additional dimension beyond the standard RGB image space. This canonical space serves as an intermediate representation that captures essential facial attributes (head posture, eye movement, eyebrow movement, mouth movement) in a structured manner, enabling fine-grained control while maintaining implementation feasibility through the two-stage motion transfer process
Solution Approach 2:
The patent employs a canonical code as an intermediary between the source image and the driving images. This canonical code acts as a mediator that decouples the complex motion control into manageable components (head posture, eyes, eyebrows, mouth) while being derived from simple RGB inputs, thus bridging the gap between simple implementation and detailed control
2Ease of operation
If methods use facial keypoints for generating facial expressions, then detailed facial attribute control is achieved, but additional supervision is required which limits real-world application
Solution Approach 1:
The patent extracts essential facial motion information directly from the RGB driving images and audio signals without requiring explicit facial keypoint annotations. The canonical encoder automatically learns and extracts the necessary motion patterns (head posture, eye movement, eyebrow movement, mouth movement) from the input data, eliminating the need for additional keypoint supervision while maintaining detailed facial attribute control
Solution Approach 2:
The system performs self-supervised learning where the canonical encoder automatically adapts to extract meaningful facial attributes from the driving images and audio without external keypoint guidance. The model serves itself by learning the canonical representation directly from the available RGB and audio inputs, making the system applicable to real-world scenarios where annotated keypoints are unavailable
3Stability of the object's composition
If methods generate talking face with similar movement and identity from target video, then identity preservation is improved, but detailed facial attribute control such as eye movement is difficult
Solution Approach 1:
The patent segments the facial motion control into distinct components through the canonical space representation: head posture, eye movement, eyebrow movement, and mouth movement. Each component can be independently controlled and adjusted while preserving the overall identity characteristics of the source face, enabling detailed facial attribute control without losing identity preservation
Data Source
AI summary
Provided are face image generation method and device for generating a controllable talking face image. The method includes: a face image generation method for generating a controllable talking face image, the method comprising: receiving a source image and a series of driving images, sampled from the same video, and input audio; acquiring a style latent code including a source latent code and a driving latent code by encoding the source image and the series of driving images into a visual space by a visual encoder; acquiring an audio feature including an audio latent code by encoding the input audio by an audio encoder; acquiring a canonical code by mapping the source latent code to a canonical space by a canonical encoder; acquiring a motion code by combining the driving latent code with the audio latent code, and mapping the combined code to a multimodal motion space by a multimodal motion encoder; acquiring a multimodal fused latent code by combining the canonical code with the motion code; and generating a talking face image by transferring the multimodal fused latent code to a generative adversarial network (GAN).