Processing-in-Memory Architecture for Transformer Multi-Head Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing transformer-based neural networks face inefficiencies in performing multi-head attention operations due to the high computational demands and resource utilization, particularly in natural language processing tasks.
Innovation Solution
Implementing a neural network architecture that utilizes processing-in-memory (PIM) devices with memory banks and processing units to perform multi-head attention operations, including specific storage and computation methods for key and value vectors, leveraging PIM devices to enhance efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional von Neumann architecture is used for transformer multi-head attention, then programming simplicity is maintained, but computational efficiency and resource utilization deteriorate due to high computational demands
Solution Approach 1:
The system segments the transformer multi-head attention computation into multiple independent PIM devices, each handling specific head computations. Each PIM device contains dedicated memory banks for storing key vectors and value vectors, allowing parallel processing of different attention heads simultaneously, thereby improving computational efficiency while distributing complexity across multiple simpler units
Solution Approach 2:
A PIM controller acts as an intermediary between the host processor and the PIM devices, managing data transfers and coordination. The controller receives query vectors from the host, distributes them to appropriate PIM devices, and collects results, simplifying the overall system architecture while enabling efficient parallel computation
2Measurement precision
If multi-head attention operations are performed with high computational demands, then processing accuracy is improved, but resource utilization deteriorates
Solution Approach 1:
The patent merges storage and computation functions into the PIM devices themselves. Memory banks within each PIM device store key and value vectors, and processing units perform matrix-vector multiplications directly in-memory, eliminating the need to transfer data between separate memory and processing units, thereby reducing energy consumption while maintaining computational accuracy
Solution Approach 2:
The system transitions from sequential processing to parallel processing by introducing multiple PIM devices that operate simultaneously. Each device handles different attention heads in parallel, effectively adding a temporal dimension to the computation and improving resource utilization while maintaining processing accuracy
3Speed
If key vectors and value vectors are stored in memory banks, then computational speed is improved, but device complexity increases
Solution Approach 1:
Memory banks are segmented and assigned to specific PIM devices based on the attention heads they support. Each PIM device has dedicated memory banks for its specific key and value vectors, enabling fast local access during computation while keeping the overall configuration manageable through systematic assignment
Solution Approach 2:
Each PIM device is self-contained with its own memory banks and processing units, capable of performing complete attention computations independently. This self-service architecture improves computational speed by eliminating data transfer bottlenecks while managing complexity through modular, autonomous units
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A neural network architecture includes a plurality of processing-in-memory (PIM) devices configured to perform multi-head attention for transformer model. Each of the plurality of PIM devices includes a plurality of memory banks configured to store key vectors and value vectors that are used as input data of the multi-head attention, and a plurality of processing units corresponding to the plurality of memory banks and configured to perform the multi-head attention via the transformer model using the key vectors and the value vectors. The plurality of memory banks are configured to store the key vectors in a first manner and store the value vectors in a second manner that is different from the first manner.