Speech Synthesis Image Motion Estimation via Global-Local Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models for motion estimation in image transformation require a large volume and numerous calculations due to the need for multiple geometric transformations, leading to inefficiencies.
Innovation Solution
A method that separates global and local motions in speech synthesis images using a first global geometric transformation predictor, a local geometric transformation predictor, and a geometric transformation combiner to reduce the model's volume and computations by estimating these motions separately.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple geometric transformations are used in parallel for accurate motion estimation, then motion estimation accuracy is improved, but model volume and computation requirements increase
Solution Approach 1:
The patent segments the motion estimation process into two distinct components: global geometric transformation (handling overall body motion) and local geometric transformation (handling facial expressions and speech-related motions). This segmentation allows each component to be optimized independently, reducing the overall model complexity while maintaining accurate motion estimation for both global and local movements.
2Measurement precision
If multiple geometric transformations are used in parallel for accurate motion estimation, then motion estimation accuracy is improved, but computation requirements increase
Solution Approach 1:
By dividing the transformation process into global and local components, the computational burden is distributed and optimized separately. The global transformation handles large-scale motions with simpler computations, while the local transformation focuses computation only on facial regions, significantly reducing overall computational requirements compared to applying multiple transformations across the entire image.
Solution Approach 2:
The patent applies different transformation strategies to different regions of the image: global geometric transformation is applied to the entire image for overall motion, while local geometric transformation is applied only to facial regions for speech and expression-related motions. This local quality approach optimizes computation by avoiding unnecessary transformations in non-critical regions.
Data Source
AI summary
An apparatus according to an embodiment is a speech synthesis image generating apparatus based on machine learning. The apparatus includes a first global geometric transformation predictor to receive a source image and a target image including the same person, and predict a global geometric transformation for a global motion of the person between the source image and the target image based on the source image and the target image, a local geometric transformation predictor to predict a local geometric transformation for a local motion of the person between the source image and the target image based on preset input data, a geometric transformation combiner to calculate a full motion geometric transformation for a full motion of the person by combining the global geometric transformation and the local geometric transformation, and an image generator to reconstruct the target image based on the source image and the full motion geometric transformation.