Adaptive Sparse Attention Pattern for Transformer Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current transformer models with fixed sparse attention patterns lack adaptability and effectiveness in identifying task-specific important tokens, leading to suboptimal prediction accuracy and requiring additional training resources.
Innovation Solution
An adaptive sparse attention pattern is developed that identifies and includes only the most important tokens for each layer and task, eliminating the need for pre-training and reducing training steps by adding the sparse attention pattern to a pre-trained model, thereby enhancing prediction accuracy without increasing computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a fixed sparse attention pattern is used to reduce computation time and memory, then computational efficiency is improved, but prediction accuracy deteriorates due to inability to identify task-specific important tokens
Solution Approach 1:
The patent transforms the static fixed sparse attention pattern into a dynamic adaptive pattern that can adjust to different tasks. The system learns task-specific important tokens during fine-tuning and dynamically constructs sparse attention patterns based on these learned tokens, allowing the attention mask to adapt to different downstream tasks while maintaining computational efficiency.
Solution Approach 2:
The patent changes the parameters of the attention mechanism by learning task-specific token importance weights during fine-tuning. Instead of using a fixed sparsity pattern, the system learns which tokens are important for each task and adjusts the attention mask parameters accordingly, thereby improving prediction accuracy while maintaining the benefits of sparsity.
2Device complexity
If a fixed sparse attention pattern is applied to all layers, then device complexity is reduced, but adaptability deteriorates due to lack of layer-specific customization
Solution Approach 1:
The patent segments the attention mechanism by layer, allowing each transformer layer to have its own task-specific sparse attention pattern. During fine-tuning, the system learns important tokens separately for each layer, enabling layer-specific adaptation to the downstream task while maintaining overall system simplicity through the unified fine-tuning process.
3Quantity of substance
If pre-training is performed with sparse attention pattern, then training resources are consumed, but adaptability deteriorates due to inability to learn task-specific token importance
Solution Approach 1:
The patent performs preliminary pre-training with full attention pattern to establish a strong baseline model, then applies sparse attention pattern during fine-tuning stage. This preliminary full-attention pre-training provides a solid foundation that enables the subsequent sparse attention fine-tuning to effectively learn task-specific token importance without requiring additional pre-training resources.
Solution Approach 2:
The patent inverts the conventional approach by first using full attention during pre-training and then applying sparse attention during fine-tuning, rather than using sparse attention from the beginning. This inversion allows the model to first learn comprehensive representations with full attention, then adapt to task-specific patterns with sparse attention, improving both resource efficiency and adaptability.
Data Source
AI summary
The technology described herein is directed to an adaptive sparse attention pattern that is learned during fine-tuning and deployed in a machine-learning model. In aspects, a row or a column in an attention matrix with an importance score for a task that is above a threshold importance score is identified. The important row or the column is included in an adaptive attention pattern used with a machine-learning model having a self-attention operation. In response to an input, a task-specific inference is generated for the input using the machine-learning model with the adaptive attention pattern.