Phonetic-Sequence Language Model Training for Speech Error Robustness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language models like BERT require costly speech processing for pre-training and are heavily dependent on speech synthesizers and recognizers, necessitating re-training when these components change, making them impractical for large-scale applications.
Innovation Solution
A language model training device that converts natural language text to phonetic letters and adds noise to generate training data, allowing pre-training and fine-tuning independent of speech synthesis and recognition, using morphological analysis and noise-adding techniques to simulate speech recognition errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If speech processing including speech synthesis and speech recognition is used to prepare training data, then the language model can be trained to be robust against speech recognition errors, but the computational cost becomes prohibitively high
Solution Approach 1:
The patent creates artificial noisy speech data by copying clean speech waveforms and adding synthesized noise signals instead of using expensive speech-to-text processing pipelines. This copying approach generates sufficient training variability at fraction of the computational cost while maintaining the ability to teach robustness against recognition errors.
Solution Approach 2:
The patent replaces the mechanical speech processing system (speech synthesis → noise addition → speech recognition) with a direct signal processing approach where noise is added to speech waveforms in the time or frequency domain, eliminating the need for computationally intensive speech-to-text conversion while achieving similar training objectives.
2Ease of manufacture
If speech synthesis and speech recognition are used for pre-training, then training data can be generated, but the system becomes heavily dependent on the specific speech synthesizer and recognizer used
Solution Approach 1:
The patent introduces an intermediary noise addition module that operates on speech waveforms directly without involving speech synthesizers or recognizers. This intermediary component generates noisy training data in a universal manner that is independent of any specific speech system, allowing the language model to be trained once and deployed across multiple different speech recognition systems.
Solution Approach 2:
The patent extracts the essential training objective (teaching robustness to noise) from the expensive speech processing pipeline and implements it through direct signal processing operations. By taking out the speech synthesis and recognition components and replacing them with universal noise addition techniques, the system achieves the same training goal without system-specific dependencies.
3Measurement precision
If billions of sentences are processed for pre-training large-scale language models, then high performance is achieved, but the computational resources required become impractical when using expensive speech processing
Solution Approach 1:
The patent replaces the expensive mechanical speech processing system with efficient signal processing operations that can handle billions of training examples. By substituting speech-to-text conversion with direct noise addition to waveforms, the system achieves linear scaling efficiency that makes billion-sentence pre-training practical while maintaining high language model performance.
Solution Approach 2:
The patent changes the parameter space from text-domain processing to signal-domain processing, where noise can be added efficiently in the time or frequency domain. This parameter change enables processing of vastly larger datasets at lower computational cost, making billion-sentence pre-training feasible while achieving high model performance.
Data Source
AI summary
A language model training device, independent of speech synthesis and speech recognition performances, allowing training of a large-scale language model at low computational cost, includes: a converting means for converting natural language text to output a sequence of phonetic letters; and a training means for training a language model using the text and the sequence of phonetic letters output from the converting means.