Speech-Text Synchronization Using UNet Pronunciation Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for synchronizing text and speech in audio signals, such as dynamic time warping and Viterbi algorithms, struggle with alignment performance when dealing with repeated sounds or specific words, leading to inaccuracies.
Innovation Solution
A neural network model with a UNet structure, combining CNNs and RNNs, is used to analyze local and global similarity regions, and trained with cross-entropy loss to synchronize text and speech by correlating audio and text pronunciation information, using international phonetic alphabet (IPA) tokens and cross-correlation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If forced alignment algorithms (DTW or Viterbi) are used to synchronize text and speech, then the synchronization process is computationally efficient and straightforward, but alignment performance deteriorates when specific sounds or words are repeated
Solution Approach 1:
The patent introduces an attention mechanism as an intermediary component between the audio encoder and text encoder. This attention mechanism dynamically weights the relevance of different audio frames to each text token, allowing the model to selectively focus on relevant portions of the audio signal even when sounds are repeated. The attention scores serve as adaptive mediators that resolve ambiguities caused by repeated patterns.
Solution Approach 2:
The patent transforms the fixed alignment constraints of traditional forced alignment algorithms into dynamic parameters through the attention mechanism. Instead of enforcing rigid sequential alignment, the attention weights allow flexible parameter adaptation where the model can assign different importance levels to repeated sounds based on contextual relevance, effectively changing the alignment parameters dynamically rather than maintaining fixed constraints.
2Measurement precision
If traditional forced alignment methods are used, then the system complexity remains low with straightforward implementation, but the model cannot effectively capture local and global similarities in pronunciation patterns
Solution Approach 1:
The patent segments the synchronization task into distinct functional components: audio encoding, text encoding, attention-based correlation, and timestamp generation. Each component processes specific aspects of the synchronization problem independently, with the attention mechanism specifically dedicated to capturing local and global similarities. This segmentation allows complex functionality to be achieved through modular, specialized sub-components.
Solution Approach 2:
The patent creates a composite model structure that combines multiple neural network components (audio encoder, text encoder, attention mechanism) into an integrated system. Each component contributes specific capabilities, and their combination creates a system that captures both local pronunciation similarities through attention and global structural patterns through the encoder architectures, achieving comprehensive pronunciation correlation analysis.
Data Source
AI summary
Disclosed is an audio signal processing device for synchronizing an audio signal and text with a speech signal, the audio signal including speech and the text corresponding to the speech. A processor of the audio signal processing device obtains first audio pronunciation information corresponding to the speech, the first audio pronunciation information being divided with regard to multiple frames included in the audio signal, and obtains first text pronunciation information corresponding to the text, the first text pronunciation information being divided with regard to multiple segments. The processor obtains information indicating a correlation between second audio pronunciation information, which is a feature extracted from each of the multiple frames of the first audio pronunciation information, and second text pronunciation information, which is a feature extracted from each of the multiple segments of the first text pronunciation information, and synchronizes the text with the speech signal according to the information indicating the correlation.