Conditional Maximum Entropy Classifier for Language Model Data Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language processing systems face performance degradation due to the mismatch between training and actual domain data, particularly in speech recognition, where filtering out non-dictated text is challenging using hand-crafted rules, which are costly, language-dependent, and lack generalization.
Innovation Solution
A method using a Conditional Maximum Entropy classifier to filter out non-dictated text units based on probability, employing a weighted linear combination of feature values to determine suitability for language model creation, allowing for efficient and language-independent data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hand-crafted rules are used to filter non-dictated text, then filtering can be performed, but the approach does not generalize well to different situations and languages, and developing and testing rules is very costly
Solution Approach 1:
The patent replaces hand-crafted mechanical rule-based filtering with a machine learning classifier system. The classifier is trained on labeled data to automatically distinguish between dictated and non-dictated text, eliminating the need for manual rule creation and testing while improving generalization to different languages and situations.
Solution Approach 2:
The patent changes the filtering approach from static hand-crafted rules to dynamic parameter-based classification using machine learning models. The classifier learns optimal parameters from training data, enabling adaptive filtering that generalizes well across different domains and languages without requiring manual rule adjustment.
2Quantity of substance
If all harvested data is used for LM adaptation, then more data is available for training, but the quality of data is compromised because non-dictated text corrupts the language model
Solution Approach 1:
The patent extracts and removes non-dictated text from the harvested data using a trained classifier. By identifying and separating unwanted content (headers, signatures, foreign language text) from legitimate dictated text, the system maintains high data quality while preserving the maximum amount of useful training data for language model adaptation.
Solution Approach 2:
The patent performs preliminary filtering of harvested data before it is used for language model training. The classifier pre-processes the data to remove potentially harmful non-dictated text, ensuring that only high-quality dictated text enters the training pipeline, thus preventing model corruption from the outset.
3Ease of manufacture
If hand-crafted rules are used for filtering, then filtering can be implemented, but rules are usually language dependent and porting rules from one language to another almost equals to rewriting the rules
Solution Approach 1:
The patent creates a universal filtering system using machine learning classifiers that can process multiple languages with a single model. The classifier learns language-agnostic features from training data, enabling it to filter non-dictated text effectively across different languages without requiring separate rule sets for each language.
Data Source
AI summary
The method and apparatus utilize a filter to remove a variety of non-dictated words from data based on probability and improve the effectiveness of creating a language model.