Mouth Shape Correction Model Preserving Facial Expressions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies for mouth shape correction in digital human actors often result in the alteration of true facial expressions, failing to effectively correct mouth shape defects without affecting the actor's genuine expressions, which is undesirable in video presentations.
Innovation Solution
A mouth shape correction model comprising a mouth feature extraction module, a key point extraction module, a first video module, a second video module, and a discriminator, which extracts and splices features to generate a predicted face image that corrects mouth shape defects while preserving the actor's true expressions, using a training method that aligns video and audio data for improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning-based mouth shape transition is used, then mouth shape transfer accuracy is improved, but mouth shape defects are reproduced and true expressions are lost
Solution Approach 1:
The patent segments the mouth region from the rest of the face by using a mask to extract only mouth-related features. This allows the model to focus specifically on mouth shape correction without being influenced by other facial features, thereby correcting mouth defects while preserving true facial expressions.
Solution Approach 2:
The patent extracts mouth shape features separately from the overall facial expression features. By isolating and processing mouth features independently, the system can correct mouth shape defects without altering the actor's genuine facial expressions, resolving the contradiction between accuracy and expression preservation.
2Manufacturing precision
If AI expression driving technology is used for mouth shape correction, then mouth shape accuracy is improved, but true facial expressions are greatly affected
Solution Approach 1:
The patent applies local quality by treating the mouth region differently from the rest of the face. It extracts and processes mouth features with higher precision while maintaining the original quality of other facial features, thus achieving accurate mouth correction without affecting true facial expressions.
Solution Approach 2:
The patent segments facial features into mouth-specific features and other facial features, processing them through different pathways. This segmentation allows independent optimization of mouth shape correction while preserving the integrity of other facial expressions.
3Reliability
If mouth shape correction is applied, then mouth shape defects are reduced, but other facial expressions may be altered
Solution Approach 1:
The patent introduces an intermediary mechanism - a dedicated mouth feature extraction and processing module that acts as a mediator between the input video and the corrected output. This intermediary ensures that only mouth-related modifications are applied while other facial features pass through unchanged, maintaining facial expression integrity.
Solution Approach 2:
The patent applies local quality enhancements specifically to the mouth region while leaving other facial features untouched. This localized processing approach improves mouth shape reliability without compromising the overall stability and integrity of facial expressions.
Data Source
AI summary
Embodiments of this disclosure provide a mouth shape correction model, and model training and application methods. The model includes a mouth feature extraction module, a key point extraction module, a first video module, a second video module, and a discriminator. The training method includes: based on a first original video and a second original video, extracting corresponding features by using various modules in the model to train the model; and when the model meets a convergence condition, completing the training to generate a target mouth shape correction model. The application method includes: inputting a video in which a mouth shape of a digital-human actor is to be corrected and corresponding audio into a mouth shape correction model, to obtain a video in which the mouth shape of the digital-human actor in the video is corrected, wherein the mouth shape correction model is a model trained by using the training method.