N-gram Selection for Language Model Size Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Higher-order N-gram language models become impractically large during training, leading to storage and efficiency issues, while existing pruning techniques significantly reduce model quality.
Innovation Solution
Implementing significance-based N-gram selection and modified weighted difference pruning to control the size of the N-gram model by excluding higher-order estimates unless justified by training data and using iterative backoff weight computation to maintain model quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If higher-order N-gram models are used to improve language model quality, then perplexity decreases and model quality improves, but model size becomes impractically large
Solution Approach 1:
The patent extracts and removes redundant or low-value N-grams from the model. Specifically, it applies pruning techniques that identify and eliminate N-grams that do not contribute significantly to model quality, thereby reducing model size while preserving the essential language modeling capabilities that achieve low perplexity on human-generated text.
Solution Approach 2:
The patent applies different treatment to different N-grams based on their individual characteristics. Rather than uniformly pruning all high-order N-grams, the system evaluates each N-gram's contribution to model quality and selectively retains only those that provide meaningful improvements, allowing local optimization of model composition.
2Quantity of substance
If existing pruning techniques are applied to reduce model size, then model size decreases, but model quality significantly deteriorates
Solution Approach 1:
The patent employs feedback mechanisms during the pruning process by evaluating model performance metrics (such as perplexity) at different stages of N-gram removal. This allows the system to identify the optimal pruning point where model size is reduced without causing significant quality deterioration, using the performance feedback to guide further pruning decisions.
Solution Approach 2:
The patent changes the parameters governing which N-grams are retained in the model. By adjusting selection criteria and retention thresholds, the system optimizes the balance between model size and quality, keeping N-grams that provide the most value while discarding those that contribute minimally, thereby achieving size reduction without significant perplexity increase.
Data Source
AI summary
Described is a technology by which a statistical N-gram (e.g., language) model is trained using an N-gram selection technique that helps reduce the size of the final N-gram model. During training, a higher-order probability estimate for an N-gram is only added to the model when the training data justifies adding the estimate. To this end, if a backoff probability estimate is within a maximum likelihood set determined by that N-gram and the N-gram's associated context, or is between the higher-order estimate and the maximum likelihood set, then the higher-order estimate is not included in the model. The backoff probability estimate may be determined via an iterative process such that the backoff probability estimate is based on the final model rather than any lower-order model. Also described is additional pruning referred to as modified weighted difference pruning.