Task-Specific Sparse Attention Module for Few-Shot Continual Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models experience catastrophic forgetting when trained on new tasks, especially under few-shot learning paradigms, leading to significant decreases in accuracy for previously learned tasks due to rapid parameter changes.
Innovation Solution
The implementation of a task-specific sparse attention module within a transformer neural network architecture, which encodes input tokens into queries, keys, and values to compartmentalize tasks into sub-units, allowing selective activation and optimization between current and previous tasks through inner and outer loops, enabling the model to retain knowledge across tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the parameters of the machine learning model are rapidly changed to adapt to limited data for each new task under few-shot learning, then the model can quickly adapt to new tasks, but the model experiences accelerated catastrophic forgetting of previously learned tasks
Solution Approach 1:
The patent segments the model parameters into task-specific parameters and shared parameters. Task-specific parameters are rapidly updated for each new task using few-shot data, while shared parameters remain relatively stable to preserve previously learned knowledge. This segmentation allows the model to adapt to new tasks without completely overwriting prior learning, thus reducing catastrophic forgetting.
Solution Approach 2:
The patent applies parameter changes by selectively updating only certain parameters (task-specific) while keeping others (shared) fixed or slowly adapting. This selective parameter update strategy enables quick adaptation to new tasks while maintaining stability in previously learned representations, directly addressing the contradiction between adaptability and retention.
2Productivity
If the machine learning model is trained on a sequence of tasks with few samples per task, then the training efficiency is improved, but the model loses previously acquired knowledge more quickly
Solution Approach 1:
By segmenting parameters into task-specific and shared components, the model can efficiently learn from few samples per task by updating only task-specific parameters, while shared parameters retain general knowledge across tasks. This reduces information loss while maintaining high training efficiency.
Solution Approach 2:
The shared parameters serve multiple tasks simultaneously, acting as a universal representation that captures common patterns across different tasks. This multi-functionality allows the model to retain previously learned knowledge while adapting to new tasks with few samples, reducing information loss without sacrificing training efficiency.
3Adaptability or versatility
If the parameters of the machine learning model are overwritten by the most recently learned concepts, then the model can learn new tasks effectively, but the model forgets earlier learned patterns
Solution Approach 1:
The patent divides parameters into task-specific parameters that are overwritten for each new task and shared parameters that are preserved. This segmentation ensures that new task learning does not completely overwrite earlier learned patterns, as the shared parameters maintain a repository of previously acquired knowledge.
Solution Approach 2:
The patent extracts and isolates the task-specific components that need to be updated for each new task, separating them from the shared knowledge representation. By taking out only the necessary parameters for updating, the model can learn new tasks effectively while preserving earlier learned patterns in the shared parameters.
Data Source
AI summary
One embodiment of the present invention sets forth a technique for executing a transformer neural network. The technique includes executing a first attention unit included in the transformer neural network to convert a first input token into a first query, a first key, and a first plurality of values, where each value included in the first plurality of values represents a sub-task associated with the transformer neural network. The technique also includes computing a first plurality of outputs associated with the first input token based on the first query, the first key, and the first plurality of values. The technique further includes performing a task associated with an input corresponding to the first input token based on the first input token and the first plurality of outputs.