Multi-step Language Model Interpolation via EM Weight Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language model training methods, which rely on perplexity for optimizing weights, are inadequate when dealing with out-of-domain text and lack reliability, as they do not effectively evaluate recognition accuracy, which requires multiple iterations and is computationally costly.
Innovation Solution
A multi-step interpolation method using the Expectation Maximization algorithm to estimate interpolation weights based on perplexity and classify language models into sets, followed by estimating hyper interpolation weights based on specific metrics like Character Error Rate or Word Error Rate, to generate a final language model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If perplexity-based weight optimization is used, then computation speed is improved, but reliability for out-of-domain text deteriorates
Solution Approach 1:
The patent segments the weight optimization process into two distinct stages: (1) initial weight estimation using perplexity-based EM algorithm for fast computation, and (2) fine-tuning using recognition accuracy metrics for reliability. This segmentation allows each method to be applied where it is most effective, resolving the contradiction between speed and reliability.
Solution Approach 2:
The patent applies preliminary action by first performing perplexity-based weight estimation as a preparatory step before conducting the more accurate but computationally intensive recognition accuracy optimization. This preliminary estimation provides a good starting point that reduces the computational burden of the subsequent fine-tuning phase.
2Reliability
If recognition accuracy-based weight optimization is used, then reliability is improved, but computation cost increases
Solution Approach 1:
The patent applies partial action by using recognition accuracy metrics only for the final fine-tuning phase rather than for the entire optimization process. The majority of the optimization is performed using the faster perplexity-based method, with recognition accuracy applied partially to achieve the necessary reliability without excessive computational cost.
Solution Approach 2:
By performing perplexity-based optimization as a preliminary step, the patent reduces the computational burden of the subsequent recognition accuracy optimization. The preliminary estimation provides a good starting point that requires fewer iterations of the computationally intensive decoding process.
3Device complexity
If single-step interpolation is used, then device complexity is reduced, but manufacturing precision of language model performance deteriorates
Solution Approach 1:
The patent segments the interpolation process into two distinct steps: (1) initial interpolation using perplexity-based weights, and (2) final interpolation using recognition accuracy-based weights. This multi-step segmentation achieves higher precision in language model performance while keeping each individual step relatively simple and manageable.
Solution Approach 2:
The patent applies dynamics by making the interpolation process adaptive and multi-phase rather than static and single-phase. The system dynamically transitions from a fast preliminary interpolation to a more precise final interpolation, allowing the complexity to be distributed across time and phases rather than concentrated in a single complex operation.
Data Source
AI summary
A computer-implemented method is provided for generating a language model for an application. The method includes estimating interpolation weights of each of a plurality of language models according to an Expectation Maximization (EM) algorithm based on a first metric. The method further includes classifying the plurality of language models into two or more sets based on characteristics of the two or more sets. The method also includes estimating a hyper interpolation weight for the two or more sets based on a second metric specific to the application. The method additionally includes interpolating the plurality of language models using the interpolation weights and the hyper interpolation weight to generate a final language model.