On-device NLU Continual Learning via Model Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current natural language understanding (NLU) models struggle to adapt to new user expressions and intents over time, as retraining them is resource-intensive and often requires access to sensitive user data, leading to issues like 'catastrophic forgetting' and privacy concerns.
Innovation Solution
Implementing on-device lightweight continual learning, where a base NLU model is expanded using model expansions trained on user-specific data without modifying the base model, allowing for continuous learning and adaptation on the user's device without losing previous knowledge.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the NLU model is retrained to learn new user expressions and intents, then the model's adaptability improves, but the computational resources and time required increase significantly
Solution Approach 1:
The patent segments the NLU model into a base model and multiple model expansions. Each model expansion is trained independently on specific user data to capture new expressions and intents, while the base model retains general NLU capabilities. This segmentation allows incremental updates without retraining the entire model, significantly reducing retraining time and computational resources.
Solution Approach 2:
The patent performs preliminary action by pre-training the base model with general NLU data before deployment. This preliminary training establishes a solid foundation that can handle common NLU tasks, allowing the system to focus computational resources only on learning new user-specific expressions through model expansions, rather than retraining the entire model from scratch.
2Adaptability or versatility
If the NLU model is retrained with user data, then the model learns new knowledge, but catastrophic forgetting of previous knowledge occurs
Solution Approach 1:
The patent segments the model training process into separate components: a frozen base model that preserves general NLU knowledge and multiple model expansions that learn user-specific knowledge. By training model expansions independently without modifying base model parameters, the system avoids catastrophic forgetting while enabling continuous learning of new user expressions and intents.
Solution Approach 2:
The patent introduces model expansions as intermediary components between the base model and user data. These expansions act as buffers that capture user-specific knowledge without directly modifying the base model's parameters, thereby preventing the loss of previously learned general NLU capabilities while enabling adaptation to new user expressions.
3Adaptability or versatility
If the NLU model is retrained frequently to stay updated, then the model remains current with user needs, but the computational cost and energy consumption increase
Solution Approach 1:
The patent segments the model into a static base model and dynamic model expansions. This segmentation allows the system to update only the lightweight model expansions with user data while keeping the energy-intensive base model frozen, dramatically reducing computational energy consumption for frequent updates while maintaining currentness with user needs.
Solution Approach 2:
The patent creates lightweight model expansions that are computationally inexpensive to train and update. These expansions serve as disposable, low-cost components that can be frequently updated with new user data without significant energy investment, enabling the system to stay current with user needs while minimizing computational energy consumption.
4Adaptability or versatility
If the base model parameters are modified to learn new user data, then the model adapts to user-specific patterns, but the general NLU capabilities deteriorate
Solution Approach 1:
The patent segments the model parameters into base model parameters (frozen to preserve general NLU capabilities) and model expansion parameters (trained on user data for user-specific adaptation). This parameter segmentation allows the system to achieve user-specific adaptation through model expansions while maintaining the reliability of general NLU capabilities in the base model.
Solution Approach 2:
The patent introduces model expansions as intermediary layers that handle user-specific adaptation without directly modifying base model parameters. These expansions mediate between user data and the base model, allowing user-specific pattern learning while preserving the integrity and reliability of general NLU capabilities in the base model.
Data Source
AI summary
A method includes obtaining, using at least one processor of an electronic device, a base model trained to perform natural language understanding. The method also includes generating, using the at least one processor, a first model expansion based on knowledge from the base model. The method further includes training, using the at least one processor, the first model expansion based on first utterances without modifying parameters of the base model. The method also includes receiving, using the at least one processor, an additional utterance from a user. In addition, the method includes determining, using the at least one processor, a meaning of the additional utterance using the base model and the first model expansion.