Transformer Accelerator Using Implicit Weights to Reduce Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer neural networks require a large number of weights, leading to frequent external memory access and high power consumption, making them unsuitable for mobile devices, and existing solutions do not effectively address this issue for advanced tasks like language translation and question answering.
Innovation Solution
An AI neural network accelerator that reduces weight requirements by using a reduced model for primary predictions and secondary predictions only when accuracy falls below a threshold, along with implicit weight generation based on kernel location information, reducing external memory access and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transformer neural network uses full weights for accurate predictions, then prediction accuracy is maintained, but external memory access frequency increases and power consumption rises
Solution Approach 1:
The patent segments the prediction process into two distinct phases: primary prediction using a reduced model with fewer weights, and secondary prediction using the full model only when needed. This segmentation allows the system to operate efficiently most of the time while maintaining accuracy when required. The controller dynamically switches between these two modes based on confidence thresholds or error conditions.
Solution Approach 2:
The patent applies partial action by using only a subset of the full weights (reduced model) for primary predictions. Instead of loading and processing all weights every time, the system uses a compressed representation that requires fewer weights, reducing memory access and power consumption. The full model is invoked partially only when the reduced model's confidence is insufficient.
2Adaptability or versatility
If transformer neural network loads all weights into external memory, then complete model functionality is available, but memory access time and power consumption increase
Solution Approach 1:
The patent extracts only the essential weight information needed for primary predictions into a reduced model that can be stored in on-chip memory. By separating the critical weight parameters from the complete model weights, the system achieves fast access for common operations while the full model remains available in external memory for secondary predictions when higher accuracy is needed.
3Use of energy by moving object
If reduced model is used for primary prediction, then weight requirement and external memory access are reduced, but prediction accuracy may decrease
Solution Approach 1:
The patent implements a feedback mechanism where the controller monitors the confidence or accuracy of primary predictions made by the reduced model. When the prediction confidence falls below a threshold or when uncertainty is detected, the system automatically triggers a secondary prediction using the full model. This feedback loop ensures that accuracy is maintained dynamically based on the specific prediction context.
Solution Approach 2:
The patent performs preliminary action by first attempting prediction with the reduced model before considering the full model. This preliminary step filters out many predictions that can be accurately handled by the lighter model, and only cases requiring higher accuracy proceed to the secondary prediction phase, thus optimizing the overall system performance.
4Productivity
If full transformer model is deployed for advanced tasks, then task performance is adequate, but weight sparsity is insufficient and power consumption remains high
Solution Approach 1:
The patent introduces dynamics into the system by making the model complexity adaptive rather than static. The controller dynamically selects between the reduced model and full model based on the specific task requirements, input characteristics, and confidence levels. This dynamic adaptation allows the system to use minimal resources for simple tasks while scaling up to full model capacity when task performance demands it, optimizing the energy-performance tradeoff.
Data Source
AI summary
An AI neural network accelerator for a transformer neural network includes a calculator configured to predict output tokens in units of input tokens and including a plurality of transformer operation cores operating based on a transformer model using n weights (n being a natural number), a weight generator including a weight embedding logic generated in advance as a result of training by matching weights of the transformer neural network with a×b kernel location information of the transformer neural network, and configured to generate an implicit weight based on location information of a kernel input from an external memory, and a controller configured to control an operation of each of the transformer operation cores after determining a size of the transformer model based on a number of weights.