Intent Classification Calibration via Distribution Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Intent classification models, such as those used in chat-bots, often suffer from inaccuracies due to divergent distributions between training and post-training data, leading to misclassification of input samples.
Innovation Solution
A method that adjusts the intent classification model by modifying the probability distribution based on a first relative frequency distribution of training data and a second relative frequency distribution of post-training data, using adjustment factors to generate a modified set of probabilities, allowing the model to adapt to real-world data without the need for re-training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the intent classification model uses training data distribution for classification, then the model structure remains simple, but the classification accuracy deteriorates when post-training data distribution diverges from training data distribution
Solution Approach 1:
The patent introduces adjustment factors as an intermediary component between the trained classifier and the final probability output. These adjustment factors are derived from the divergence between training data distribution and post-training data distribution, and they modify the classifier's probability estimates without changing the classifier structure itself. This mediator approach resolves the contradiction by improving reliability through distribution alignment while keeping the model complexity low.
Solution Approach 2:
The patent changes the parameters (adjustment factors) based on data distribution divergence rather than changing the model structure. By computing adjustment factors from the difference between training and post-training data distributions, and applying these factors to modify probability estimates, the system adapts to distribution shifts while maintaining the original simple classifier architecture.
2Reliability
If the model is re-trained with post-training data to adapt to real-world distribution, then classification accuracy improves, but the time and effort required increases significantly
Solution Approach 1:
The patent performs preliminary computation of adjustment factors from data distribution statistics rather than performing full re-training. By pre-computing the divergence between training and post-training distributions and converting this into adjustment factors, the system prepares the necessary adaptation parameters in advance, avoiding the time-consuming re-training process while still achieving accurate classification.
Solution Approach 2:
Instead of creating a new trained model through re-training, the patent creates a modified version of the original classifier by copying its structure and applying adjustment factors to its probability outputs. This copying approach with minimal modifications achieves adaptation to post-training data distribution without the substantial time investment required for full re-training.
3Reliability
If the model continuously adapts to new data distribution, then the classification reliability improves, but the computational resources required increase
Solution Approach 1:
The patent applies partial action by computing adjustment factors based only on the essential divergence between data distributions rather than performing complete re-training or complex adaptation procedures. This partial approach—focusing only on the necessary probability adjustments—achieves improved classification reliability while consuming minimal computational energy.
Data Source
AI summary
A method includes determining, based on an input data sample, a set of probabilities. Each probability of the set of probabilities is associated with a respective label of a set of labels. A particular probability associated with a particular label indicates an estimated likelihood that the input data sample is associated with the particular label. The method includes modifying the set of probabilities based on a set of adjustment factors to generate a modified set of probabilities. The set of adjustment factors is based on a first relative frequency distribution and a second relative frequency distribution. The first relative frequency distribution indicates for each label of the set of labels, a frequency of occurrence of the label among training data. The second relative frequency distribution indicates for each label of the set of labels, a frequency of occurrence of the label among post-training data provided to the trained classifier.