Speech Transducer Regularization Using Noisy View Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Consistency regularization for transducer-based speech-to-text models is complex due to the complexity of output distributions, and existing methods optimized with cross entropy loss are limited, leading to potential decreases in model performance.
Innovation Solution
A transducer model is trained using a training dataset with audio frames and ground truth subword unit transcriptions, generating multiple noisy views to predict probability distributions for frame synchronization decoding, and modified using transducer loss, incorporating weighted transducer divergence losses to stabilize training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If consistency regularization is applied to transducer-based speech-to-text models, then training stability is improved, but computational complexity increases due to the complexity of output distributions
Solution Approach 1:
The patent extracts the consistency regularization objective from the complex transducer output distribution by focusing on the alignment between noisy views through frame synchronization. This separates the consistency learning task from the full transducer decoding complexity, enabling stable training without requiring computation of all possible output distributions.
Solution Approach 2:
The patent segments the consistency regularization into discrete frame-level synchronization checks rather than treating the entire output distribution at once. By breaking down the continuity requirement into individual frame alignments, the computational burden is reduced while maintaining training stability.
2Ease of manufacture
If cross entropy loss is used for optimization, then training process is simplified, but model performance decreases
Solution Approach 1:
The patent introduces frame synchronization as an intermediary mechanism that bridges the simplified cross-entropy loss optimization and the complex transducer output distributions. This intermediary allows the model to achieve better performance by enforcing alignment constraints between noisy views without requiring complex loss functions.
Solution Approach 2:
The patent changes the optimization parameters by incorporating frame synchronization constraints into the training objective. This modifies how the model learns from noisy views, enabling better performance while maintaining the simplicity of cross-entropy loss optimization through the added synchronization regularizer.
3Reliability
If multiple noisy views are generated and processed, then model robustness is improved, but processing time increases
Solution Approach 1:
The patent applies preliminary action by generating multiple noisy views during the training phase rather than during inference. This allows the model to learn robust representations from augmented data during training, and then process new inputs more efficiently during deployment without the computational burden of generating multiple views for each test sample.
Data Source
AI summary
A method includes receiving an audio input and providing the audio input to a transducer model. The method also includes predicting, using the transducer model, text associated with the audio input and outputting the predicted text. The transducer model is trained using a training dataset that includes samples including audio frames and a ground truth subword unit transcription. A first view of a training sample is generated by adding first random noise to the audio frames. A second view of the training sample is generated by adding second random noise to the audio frames. The first view and the second view are provided to the transducer model. The transducer model predicts a probability distribution for each of the first view and the second view. The probability distribution includes probabilities of possible frame synchronization decoding between the audio frames and transcriptions. The transducer model is modified based on a transducer loss.