Attention-Based AI Model Thresholding for Lower Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid increase in parameter size of Large Language Models (LLM) outpaces computing performance, particularly in memory performance, leading to bottlenecks that hinder smooth operation.
Innovation Solution
Implementing a thresholding mechanism in the attention layer of an attention-based artificial intelligence model to bypass softmax and dot-product operations for attention scores or probabilities below a set threshold, reducing memory bandwidth and computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the parameter size of LLM is increased to improve language understanding and generation capabilities, then the model performance is improved, but the memory bandwidth and computational load requirements increase rapidly, creating a bottleneck in computing performance
Solution Approach 1:
The patent extracts and removes unnecessary computational operations from the attention layer by introducing a thresholding mechanism. Attention scores below a dynamically determined threshold are set to zero and excluded from subsequent computations, effectively removing redundant calculations that consume memory bandwidth and computational resources while maintaining model performance.
Solution Approach 2:
The patent changes the parameter of attention score thresholding to control the balance between performance and computational load. By dynamically adjusting the threshold based on input sequence length and maximum attention scores, the system adapts the amount of computation performed, reducing memory bandwidth requirements while preserving essential language understanding capabilities.
2Reliability
If the parameter size of LLM is increased to improve language understanding and generation capabilities, then the model performance is improved, but the computational load and memory bandwidth requirements increase, hindering smooth operation
Solution Approach 1:
The patent extracts and removes unnecessary computational operations from the attention layer by introducing a thresholding mechanism. Attention scores below a dynamically determined threshold are set to zero and excluded from subsequent computations, effectively removing redundant calculations that consume memory bandwidth and computational resources while maintaining model performance.
Solution Approach 2:
The patent applies partial action by computing and processing only the necessary portion of attention scores above the threshold, rather than processing all attention scores uniformly. This selective processing approach maintains sufficient model performance while significantly reducing computational load and improving productivity.
3Quantity of substance
If a threshold is set to bypass softmax operation for attention scores below the threshold, then memory bandwidth and computational load are reduced, but the accuracy of attention probability calculation may be affected
Solution Approach 1:
The patent implements feedback by dynamically determining the threshold based on the input sequence length and maximum attention scores in each computation. This adaptive thresholding ensures that the balance between memory bandwidth reduction and attention probability accuracy is continuously optimized based on the actual computational context.
Solution Approach 2:
The patent changes the parameter of attention score thresholding to control the balance between performance and computational load. By dynamically adjusting the threshold based on input sequence length and maximum attention scores, the system adapts the amount of computation performed, reducing memory bandwidth requirements while preserving essential language understanding capabilities.
Data Source
AI summary
A computing system that performs a process using an attention-based artificial intelligence model includes: at least one processor configured to control a process using the artificial intelligence model; and a memory configured to store instructions performed by the at least one processor, wherein, when performing a process of an attention layer, the at least one processor is configured to: obtain a query feature map matrix and a key feature map matrix from an input sequence including a plurality of tokens; obtain an attention score matrix based on a dot-product of the obtained query feature map matrix and key feature map matrix; for each of attention score vectors of the plurality of tokens included in the obtained attention score matrix, set a threshold based on a maximum value from among included attention scores; and bypass a softmax operation for an attention score less than the set threshold for each of the attention score vectors of the plurality of tokens.