Language Model Training Using Spoken and Unspeakable Corpus Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current speech recognition systems face challenges in training effective language models due to the cost and limitations of transcribed speech data, which often includes unsuitable text that dilutes the lexicon and requires large amounts of data for accurate modeling.
Innovation Solution
A computer system that collects training data from multiple sources, generates a spoken corpus from transcribed speech, filters out unspeakable text from a typed corpus to create an unspeakable corpus, and trains a classifier to distinguish between speakable and unspeakable text, thereby enhancing language modeling with discriminative data selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If transcribed speech is used as training data for language modeling, then the training data can be obtained, but the cost is high and the amount of available training data is limited
Solution Approach 1:
The patent creates a synthetic spoken corpus by copying and transforming typed text data through text-to-speech synthesis. This allows the system to generate unlimited training data from readily available typed text sources, eliminating the need to manually transcribe large amounts of speech data while maintaining the characteristics of spoken language for language model training.
2Quantity of substance
If typed text is used to supplement training data, then the quantity of training data increases, but the quality decreases due to unspeakable text diluting the lexicon
Solution Approach 1:
The patent extracts and removes unspeakable text from typed corpora using a trained classifier that identifies text unlikely to be spoken. This extraction process separates useful speakable text from harmful unspeakable text, allowing the system to supplement training data with typed text while maintaining high quality by excluding diluting content.
Solution Approach 2:
The system employs a feedback mechanism where a classifier is trained on feature vectors from both spoken and typed corpora to identify unspeakable text. This trained classifier then feeds back into the data selection process, automatically filtering typed text to retain only speakable portions for language model training, thereby maintaining quality while increasing quantity.
3Manufacturing precision
If a classifier is trained to filter typed corpus, then the quality of training data improves, but the processing time and complexity increase
Solution Approach 1:
The patent transforms typed text into feature vectors that capture linguistic characteristics relevant to speakability. By changing the representation parameters of the text data and training the classifier on these transformed features rather than raw text, the system achieves effective filtering while managing processing complexity through dimensionality reduction and feature engineering.
Data Source
AI summary
A computer system for language modeling may collect training data from one or more information sources, generate a spoken corpus containing text of transcribed speech, and generate a typed corpus containing typed text. The computer system may derive feature vectors from the spoken corpus, analyze the typed corpus to determine feature vectors representing items of typed text, and generate an unspeakable corpus by filtering the typed corpus to remove each item of typed text represented by a feature vector that is within a similarity threshold of a feature vector derived from the spoken corpus. The computer system may derive feature vectors from the unspeakable corpus and train a classifier to perform discriminative data selection for language modeling based on the feature vectors derived from the spoken corpus and the feature vectors derived from the unspeakable corpus.