Lip-Matched Voice Dubbing Using Speech-Mouth Consistency Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Professional voice dubbing is labor-intensive and expensive due to the complexities of duration and lip matching, requiring multiple human interventions that prolong the process.
Innovation Solution
A system and method that automates or assists in generating lip-matched translations and voice dubbings by using a text-to-speech synthesizer, speech-mouth consistency model, and optional frame editing and reanimation utilities to synchronize audio with video lip movements, providing automated grading and modification suggestions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If professional voice dubbing is performed manually with human translators and voice actors, then the quality of translation and lip matching is high, but the time and cost required are excessive
Solution Approach 1:
The patent replaces the manual mechanical process of human translators and voice actors with an automated system comprising a text-to-speech synthesizer and a speech-mouth consistency model. The synthesizer generates audio automatically from text, and the model evaluates and adjusts the synchronization between audio and video lip movements, eliminating the need for human intervention while maintaining high precision in lip matching.
Solution Approach 2:
The system performs self-evaluation and self-correction through the speech-mouth consistency model, which automatically assesses the synchronization between generated audio and video lip movements. The model identifies and corrects mismatches without external human feedback, enabling the system to service itself in the dubbing process.
2Adaptability or versatility
If multiple human professionals are involved in the dubbing process, then the quality and adaptability are high, but the device complexity and resource requirements increase
Solution Approach 1:
The patent creates a universal system where a single text-to-speech synthesizer can generate audio for multiple languages and contexts, and the speech-mouth consistency model can evaluate and adjust any dubbing scenario. This multi-functional approach replaces the need for multiple specialized human professionals with one versatile automated system that handles various dubbing challenges.
3Manufacturing precision
If manual duration matching is performed by adapters, then the timing precision is high, but the labor intensity and time consumption are high
Solution Approach 1:
The patent substitutes the manual duration matching performed by human adapters with an automated process where the text-to-speech synthesizer generates precise timing information from the source text, and the speech-mouth consistency model automatically adjusts and synchronizes the audio duration with the video clip without human intervention, thereby maintaining precision while dramatically improving productivity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods for generating candidate translations for use in creating synthetic or human- acted voice dubbings, aiding human translators in generating translations that match the corresponding video, automatically grading how well a candidate translation matches the corresponding video, suggesting modifications to the speed and/or timing of the translated text to improve the grading of a candidate translation, and suggesting modifications to the voice dubbing and/or video to improve the grading of a candidate translation. In that regard, the present technology may be used to fully automate the process of generating lip-matched translations and associated voice dubbings, or as an aid for human- in-the-loop processes that may reduce or eliminate the time and effort required from translators, adapters, voice actors, and/or audio editors to generate voice dubbings.