Audio-to-Video Engine Using Minimum Converted Trajectory Error
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing audio-to-video engines using maximum likelihood estimation (MLE)-based conversion processes for Gaussian Mixture Models (GMMs) often result in conversion errors that are not minimized for human perception, leading to suboptimal video generation from speech input.
Innovation Solution
The implementation of a Minimum Converted Trajectory Error (MCTE)-based process that refines the GMM by separately weighing audio and video data using a log likelihood function and applying a generalized probabilistic descent (GPD) algorithm to minimize conversion errors, thereby improving the accuracy of video feature parameter estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If MLE-based conversion process is applied to GMM, then the conversion process is simple and computationally efficient, but the converted visual trajectories do not minimize error in human perception
Solution Approach 1:
The patent segments the conversion error minimization problem into two separate weighted estimation processes: one for audio trajectories and one for visual trajectories. This segmentation allows each modality to be optimized independently with appropriate weighting, rather than treating them as a single unified MLE problem. The separate weighting enables better control over conversion errors for each trajectory type.
Solution Approach 2:
The patent introduces new parameters (weighting factors for audio and visual trajectories) that modify the standard MLE objective function. By changing the parameter structure from a single unified likelihood to a weighted sum of separate likelihoods, the system can minimize conversion errors more effectively while maintaining computational tractability through gradient-based optimization.
2Reliability
If MLE-based conversion process is used, then the training is computationally efficient, but the visual trajectory conversion does not optimize for human perception error
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing the weighted likelihood contributions from audio and visual trajectories during training. This allows the optimization process to work with pre-processed probability distributions rather than raw data, significantly reducing the computational burden during both training and inference while maintaining high visual trajectory accuracy.
Solution Approach 2:
The patent replaces the standard MLE mechanical optimization system with a weighted probabilistic framework that uses gradient ascent on a customized objective function. This substitution allows for more flexible control over the optimization process, enabling better convergence to visually accurate trajectories while managing training time through efficient gradient computation and potential parallelization.
Data Source
AI summary
Embodiments of an audio-to-video engine are disclosed. In operation, the audio-to-video engine generates facial movement (e.g., a virtual talking head) based on an input speech. The audio-to-video engine receives the input speech and recognizes the input speech as a source feature vector. The audio-to-video engine then determines a Maximum A Posterior (MAP) mixture sequence based on the source feature vector. The MAP mixture sequence may be a function of a refined Gaussian Mixture Model (GMM). The audio-to-video engine may then use the MAP to estimate video feature parameters. The video feature parameters are then interpreted as facial movement. The facial movement may be stored as data to a storage module and/or it may be displayed as video to a display device.