Masked Speech Modeling with Confidence-Guided Frame Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based ASR models tend to overfit training data, leading to difficulties in generalizing unseen data, especially when the training data is not extensive enough, and performance suffers when the domain of deployment differs from the training domain.
Innovation Solution
Implement a guided data selection method for masked speech modeling using a masking module that selects encoded representations based on confidence scores to mask relevant information, pre-training the ASR model with unlabeled speech and fine-tuning on labeled data, and applying self-supervised losses to enhance representation learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If training data volume is increased to improve ASR model accuracy, then model generalization improves, but data availability and training cost worsen
Solution Approach 1:
The patent uses speech synthesis to generate synthetic training data that copies the statistical properties of real speech data. The synthesizer creates artificial speech samples with matched phonetic distributions, allowing the model to train on expanded datasets without requiring additional real-world recording resources
Solution Approach 2:
The patent transforms the training approach by changing from using only raw real speech data to using synthesized data with controlled parameters. The speech synthesizer adjusts parameters like phonetic composition and acoustic characteristics to match target domain distributions, enabling effective training with generated data
2Measurement precision
If domain-specific training data is used to improve accuracy for specific domains, then performance on that domain improves, but adaptability to other domains worsens
Solution Approach 1:
The patent creates a universal speech synthesizer that can generate data across multiple domains by controlling phonetic composition parameters. The same synthesizer system serves multiple functions: generating domain-specific training data while maintaining the ability to adapt to different target domains through parameter adjustment rather than requiring domain-specific models
Solution Approach 2:
The patent uses parameter control in the speech synthesizer to adapt data generation to different domains. By adjusting phonetic composition parameters and acoustic characteristics, the system generates domain-appropriate synthetic data without changing the underlying model architecture, enabling both domain-specific accuracy and cross-domain adaptability
3Reliability
If more real speech data is collected to improve model performance, then training data quality improves, but privacy concerns and data collection complexity worsen
Solution Approach 1:
The patent replaces the need to collect and store sensitive real speech data with synthetic data generation. Instead of gathering actual user recordings which raises privacy concerns, the system copies the statistical and phonetic properties of speech through a synthesizer, eliminating the need for complex data collection infrastructure and privacy management systems
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A method (500) of guided data selection for masked speech modeling includes obtaining a sequence of encoded representations (211) corresponding to an utterance (100). For each respective encoded representation, the method includes processing the respective encoded representation to generate a corresponding probability distribution (414) over possible speech recognition hypotheses (412) and assigning, to the respective encode representation, a confidence score (416) as a highest probability from the corresponding probability distribution over possible speech recognition hypotheses. The method also includes selecting a set of unmasked encoded representations to mask based on the confidence scores assigned to the sequence of encoded representations. The method also includes generating a set of masked encoded representations (211m) by masking the selected set of unmasked encoded representations. Here, each masked encoded representation corresponds to a respective one of the unmasked encoded representations in the selected set of unmasked encoded representations.