Intent Detection Replay Using Prompted Concept Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing lifelong intent detection systems face challenges in generating representative samples due to reliance on class/task labels and require large volumes of old training data, leading to inefficient and costly retraining, especially in class-incremental learning scenarios.
Innovation Solution
The proposed method employs a prompt augmented generative replay (PAGeR) technique using supervised contrastive training, where only concept words are stored in memory, and a joint training approach is used with LID, LUG, SCT, and KD loss functions to generate pseudo-labeled utterances, reducing memory requirements and minimizing catastrophic forgetting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If retraining the intent detection model from scratch is performed to adapt to evolving user intents, then the model can learn new intents accurately, but the computational cost increases due to ever-increasing data volume
Solution Approach 1:
The patent segments the training process into incremental steps where the model is trained on new intents separately rather than retraining from scratch on all data. The intent detection model is updated incrementally by training on new intent data while preserving knowledge of previous intents through parameter retention and selective updating, reducing computational burden compared to full retraining.
Solution Approach 2:
The patent performs preliminary actions by storing and replaying samples from previous intents before training on new intents. A replay buffer stores historical samples that are mixed with new training data, allowing the model to preliminarily review old patterns before adapting to new ones, thus preventing catastrophic forgetting without requiring full retraining.
2Adaptability or versatility
If the intent detection model is updated to include new intents, then the model can handle new functionality, but performance on prior intents deteriorates due to catastrophic forgetting
Solution Approach 1:
The patent creates copies of historical training samples and stores them in a replay buffer. These copied samples from previous intents are replayed during training on new intents, allowing the model to maintain performance on prior intents while learning new ones. The replayed samples act as synthetic copies that preserve knowledge without requiring access to original data.
Solution Approach 2:
The patent implements feedback mechanisms where the model's performance on both old and new intents is continuously monitored. The replay buffer is dynamically adjusted based on performance feedback, and training samples are selected to maximize retention of prior knowledge while adapting to new intents. Loss functions are designed to provide feedback that balances learning new intents with maintaining performance on old ones.
3Use of energy by moving object
If generative replay is used to generate samples of old tasks, then computational cost is reduced, but the generated samples are not representative enough for effective knowledge retention
Solution Approach 1:
The patent introduces an intermediary approach by using a replay buffer that stores actual historical samples rather than relying solely on generative models to create synthetic samples. This intermediary storage mechanism preserves the authenticity and representativeness of original data while avoiding the computational overhead of generating highly realistic synthetic samples. The replay buffer acts as a mediator between old and new training data.
Solution Approach 2:
The patent changes parameters of the replay buffer dynamically, adjusting the size, sampling strategies, and composition of replayed samples based on training progress and performance metrics. By modifying parameters such as the replay ratio, buffer size, and sampling distribution, the system optimizes the balance between computational efficiency and sample representativeness without requiring complex generative models.
4Ease of manufacture
If task-specific adapters are trained in isolation for Task-IL, then each adapter can be optimized for its specific task, but erroneous task-id predictions occur due to overlap between intents from different incremental steps
Solution Approach 1:
The patent merges multiple task-specific adapters into a unified intent detection model. Instead of maintaining separate adapters for each task that require accurate task identification, the model integrates knowledge from all intents into a single framework. This merging eliminates the need for precise task-id prediction by allowing the model to handle all intents through unified parameter space and joint training objectives.
Solution Approach 2:
The patent creates a universal intent detection model that can handle multiple intents and tasks through a single framework. The model is designed to be multi-functional, accommodating both old and new intents without requiring task-specific configurations or accurate task identification. This universal approach allows the model to adapt to different intents while maintaining performance across all categories through shared representations and joint optimization.
Data Source
AI summary
Embodiments disclosed herein model lifelong intent detection as a class-incremental learning where a new set of intents/classes are added at each incremental step. To address the issue of catastrophic forgetting during lifelong intent detection (LID), an incremental learner is provided with Prompt Augmented Generative Replay, wherein unlike existing approaches that store real samples in replay memory, only concept words obtained from old intents are stored, which reduces memory consumption and speeds up incremental training still enabling not forgetting the old intents. Joint training of an incremental learner is carried out for LID and a pseudo-labeled utterance generation with objective is to classify a user utterance into one of multiple pre-defined intents by minimizing a total Loss function comprising a LID loss function, a Labeled Utterance Generation loss function, a Supervised Contrastive Training loss function, and a Knowledge Distillation loss function.