NLU Model Compression via Sparsity-Induced Regularization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Natural language understanding (NLU) models struggle with achieving desirable performance when user expressions differ from pre-collected training data, and they fail to understand new language concepts that develop over time due to their lack of transparency and explainability.
Innovation Solution
A system and method for explaining and compressing deep learning NLU models by recalculating regularization loss values and retraining the word embedding layer using an intent detection loss, which identifies and prunes unnecessary words, allowing for improved performance and explainability without additional training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If NLU models are trained using pre-collected training data, then the model can be trained with available data, but the model fails to achieve desirable performance when user expressions differ from training data and cannot understand new language concepts
Solution Approach 1:
The patent changes the parameter of word embedding vectors by introducing a regularization term that encourages sparsity. This transforms the embedding space to be more adaptable while maintaining reliability through structured constraints on the parameter changes.
Solution Approach 2:
The patent makes the NLU model dynamic by enabling it to adapt to new language concepts through iterative retraining with regularization. The model transitions from a static trained state to a dynamically adaptable system that can handle evolving user expressions.
2Adaptability or versatility
If the NLU model includes a complete word embedding layer for all possible concepts, then the model covers more language concepts, but the model size increases and cannot be deployed on personal devices
Solution Approach 1:
The patent extracts only the necessary word embeddings by applying L1 regularization that prunes unnecessary embeddings. This removes redundant parameters from the model while retaining coverage of essential language concepts, enabling deployment on resource-constrained devices.
Solution Approach 2:
The patent applies different quality standards to different parts of the embedding layer by using sparsity-inducing regularization. Important word embeddings are preserved with high quality while less important ones are pruned or compressed, creating local variations in embedding quality that optimize the overall model size-performance tradeoff.
3Adaptability or versatility
If the NLU model is made transparent and explainable to enable user feedback, then the model can improve through user input, but the complexity of implementing explanation mechanisms increases
Solution Approach 1:
The patent implements feedback mechanisms that leverage the sparsity patterns in word embeddings to explain model decisions. The regularization-induced sparsity provides natural explanations for which words are important, enabling user feedback without complex additional explanation mechanisms.
Data Source
AI summary
A method includes obtaining, using at least one processor of an electronic device, a base natural language understanding (NLU) model that includes a word embedding layer, where the word embedding layer is associated with at least one training utterance. The method also includes calculating, using the at least one processor, a regularization loss value for use in a determination of an intent detection loss, where the regularization loss value reveals an effect of word embeddings on intent determination of the training utterance. The method further includes retraining, using the at least one processor, the word embedding layer of the base NLU model using the intent detection loss to obtain a retrained NLU model.