Transformer Multi-Head Attention Using Differentiated PIM Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network architectures for transformer-based multi-head attention face challenges in efficiently performing matrix-vector multiplications and storing key and value vectors, which affects computational efficiency and memory utilization.
Innovation Solution
The proposed neural network architecture employs a plurality of processing-in-memory (PIM) devices with memory banks configured to store key and value vectors differently and processing units to perform multi-head attention via a transformer model, optimizing matrix-vector multiplications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If key vectors and value vectors are stored in the same manner in memory banks, then the storage structure is simple, but computational efficiency during multi-head attention operations is reduced
Solution Approach 1:
The patent segments the storage of key vectors and value vectors into different memory banks with different storage manners. Key vectors are stored in a first manner while value vectors are stored in a second manner, allowing optimized retrieval patterns for each vector type during multi-head attention computations
Solution Approach 2:
Different memory banks are assigned different storage configurations tailored to their specific vector types. This local optimization allows key vectors to be accessed efficiently for attention score computation while value vectors are organized for efficient weighted summation operations
2Productivity
If traditional neural network architectures are used for transformer-based multi-head attention, then implementation is straightforward, but memory utilization and computational efficiency are insufficient
Solution Approach 1:
The patent replaces traditional von Neumann architecture with processing-in-memory (PIM) devices that perform computations directly within the memory subsystem. This substitution eliminates data transfer bottlenecks between separate storage and processing units, significantly improving computational efficiency for matrix-vector multiplications in multi-head attention
Solution Approach 2:
The PIM devices are designed to perform multiple functions: storing vectors, performing matrix-vector multiplications, computing attention scores, and generating attention distributions. This multi-functionality integrates what were previously separate operations into a unified efficient process
3Reliability
If more memory is allocated for storing key and value vectors, then data availability improves, but memory requirements and system cost increase
Solution Approach 1:
Key vectors and value vectors are pre-organized in memory banks with optimized storage configurations before attention computations begin. This preliminary arrangement ensures that data is immediately available in the required formats when computations start, eliminating the need for additional memory buffers or repeated data loading
Data Source
AI summary
A neural network architecture includes a plurality of processing-in-memory (PIM) devices configured to perform multi-head attention for transformer model. Each of the plurality of PIM devices includes a plurality of memory banks configured to store key vectors and value vectors that are used as input data of the multi-head attention, and a plurality of processing units corresponding to the plurality of memory banks and configured to perform the multi-head attention via the transformer model using the key vectors and the value vectors. The plurality of memory banks are configured to store the key vectors in a first manner and store the value vectors in a second manner that is different from the first manner.