Backoff Score Calculation for Sparse N-gram Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Statistical machine translation faces challenges in efficiently generating language models due to sparse data issues and the need for accurate probability calculations, particularly in handling n-grams with zero or undefined frequencies, which affects translation quality and efficiency.
Innovation Solution
The development of a system that uses n-grams with backoff scores calculated as a function of relative frequencies, allowing for direct determination of backoff values without normalizing probabilities, and enabling distributed generation of n-gram backoff values within a shared environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional statistical machine translation uses n-grams with zero or undefined frequencies, then translation coverage is improved, but probability calculation accuracy deteriorates
Solution Approach 1:
The patent applies beforehand cushioning by pre-calculating backoff scores for n-grams based on their frequencies in training data. When an n-gram has zero or undefined frequency during translation, the system uses these pre-computed backoff scores to provide a cushion against probability calculation errors, ensuring stable and accurate translations even for rare or unseen n-grams.
Solution Approach 2:
The backoff score acts as an intermediary between the raw frequency data and the final probability calculation. Instead of directly using zero or undefined frequencies which cause calculation errors, the system introduces backoff scores as a mediating parameter that bridges the gap between sparse data and accurate probability estimation.
2Measurement precision
If the system calculates normalized probabilities for all n-grams, then translation accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent extracts the essential information needed for probability calculation from the full normalized probability distribution. By taking out only the necessary backoff score components and pre-computing them from training data, the system achieves translation accuracy without the computational burden of normalizing all n-gram probabilities during each translation operation.
Solution Approach 2:
The system performs preliminary action by pre-calculating backoff scores during the training phase rather than during translation. This allows the heavy computational work to be done once during training when data is available, while translation can proceed quickly using these pre-computed values without repeated normalization calculations.
3Reliability
If the system handles sparse data by using backoff n-grams, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent applies parameter changes by transforming the approach from using raw frequencies or normalized probabilities to using backoff scores as a new parameter. This parameter transformation allows the system to handle sparse data reliably by smoothly transitioning between different n-gram orders (e.g., from 5-grams to 4-grams) based on the availability of frequency data, while maintaining a manageable model structure.
Data Source
AI summary
Systems, methods, and computer program products for machine translation are provided. In some implementations a system is provided. The system includes a language model including a collection of n-grams from a corpus, each n-gram having a corresponding relative frequency in the corpus and an order n corresponding to a number of tokens in the n-gram, each n-gram corresponding to a backoff n-gram having an order of n−1 and a collection of backoff scores, each backoff score associated with an n-gram, the backoff score determined as a function of a backoff factor and a relative frequency of a corresponding backoff n-gram in the corpus.