Speech Synthesis Image Generation with Global-Local Motion Separation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models for motion estimation in image conversion and synthesis require a large volume and numerous calculations due to the need for multiple geometric transformations, leading to inefficiencies.
Innovation Solution
A method that separates global and local motions in speech synthesis images using a first global geometric transformation predictor, a local feature tensor predictor, and an image generator, reducing the model's volume and computation requirements by estimating these motions separately.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple geometric transformations are used in parallel for accurate motion estimation, then motion estimation accuracy is improved, but model volume and computational requirements increase
Solution Approach 1:
The patent divides motion estimation into two separate modules: a global geometric transformation module for large-scale motion and a local feature tensor module for fine-grained motion details. This segmentation allows each module to specialize in specific aspects of motion, achieving accurate motion estimation without requiring multiple general-purpose geometric transformations, thus reducing model volume while maintaining precision.
2Measurement precision
If multiple geometric transformations are used in parallel for accurate motion estimation, then motion estimation accuracy is improved, but computational requirements increase
Solution Approach 1:
By segmenting motion estimation into global and local components, the computational workload is distributed efficiently. The global geometric transformation handles coarse motion with fewer calculations, while the local feature tensor handles fine details only where needed, reducing overall computational requirements compared to applying multiple full-image geometric transformations.
Solution Approach 2:
The patent extracts and processes only the necessary motion components separately rather than applying multiple complete geometric transformations to the entire image. The local feature tensor extracts fine-grained motion details only from relevant regions, reducing computational requirements while maintaining accuracy.
3Productivity
If global and local motions are estimated separately using dedicated predictors, then computational efficiency is improved, but model structure complexity increases
Solution Approach 1:
The patent merges the outputs of the global geometric transformation predictor and the local feature tensor predictor in the image generator module. This combining approach allows the system to leverage both global and local motion information efficiently, achieving high computational efficiency while the integrated structure manages the complexity of having multiple predictors.
Data Source
AI summary
An apparatus for generating a speech synthesis image according to a disclosed embodiment is an apparatus for generating a speech synthesis image based on machine learning, the apparatus including a first global geometric transformation predictor configured to be trained to receive each of a source image and a target image including the same person, and predict a global geometric transformation for a global motion of the person between the source image and the target image based on the source image and the target image, a local feature tensor predictor configured to be trained to predict a feature tensor for a local motion of the person based on input target image-related information, and an image generator configured to be trained to reconstruct the target image based on the global geometric transformation, the source image, and the feature tensor for the local motion.