PIM Memory Architecture with Adaptive Weight Distribution for MAC Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing processing-in-memory (PIM) architectures face inefficiencies in data movement and calculation speed, particularly in decoder-based large language models, leading to increased operation times as batch_size and beam_size increase, necessitating a shift from BLAS2 to BLAS3 operations.
Innovation Solution
A memory apparatus with a weight handler circuit, MAC circuit, and register file that adjusts weight distribution based on MAC circuit specifications, using broadcasting units, asynchronous FIFO, and buffers to enable simultaneous operations across multiple MAC units, facilitating BLAS3 performance without additional latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If weights are read from memory bank for each MAC unit operation, then calculation accuracy is maintained, but operation time increases and productivity decreases
Solution Approach 1:
The weight handler circuit pre-fetches and distributes weights to multiple MAC units before the actual computation begins. By preparing the weight distribution in advance based on MAC circuit specifications, the system avoids repeated memory access during computation, thereby maintaining accuracy while improving operation speed.
Solution Approach 2:
The weight handler circuit acts as an intermediary between the memory bank and MAC units. It receives weights from memory, processes their distribution according to MAC specifications, and delivers them to appropriate MAC units, thereby eliminating direct repeated access to memory and reducing operation time.
2Productivity
If multiple MAC units operate simultaneously, then productivity increases, but device complexity increases
Solution Approach 1:
The weight handler circuit is designed with multi-functionality to handle weight distribution for multiple MAC units simultaneously. It can broadcast weights to all MAC units or selectively distribute them based on operation requirements, thereby supporting parallel operations without proportionally increasing overall system complexity.
Solution Approach 2:
The system divides the weight distribution task into segments handled by the weight handler circuit, which manages multiple MAC units independently. Each MAC unit receives appropriate weights through the segmented distribution mechanism, enabling parallel processing while keeping the control logic organized and manageable.
3Adaptability or versatility
If weight distribution is adjusted based on MAC specifications, then adaptability improves, but device complexity increases
Solution Approach 1:
The weight handler circuit implements dynamic weight distribution that adapts to different MAC circuit configurations. Based on received specifications (such as number of MAC units and operation frequency), the circuit automatically adjusts its distribution strategy, providing high adaptability while keeping the adjustment logic integrated and manageable within the same circuit.
Data Source
AI summary
A memory apparatus is configured to perform a processing-in-memory (PIM) operation. The memory apparatus includes a weight handler circuit configured to receive weights from a memory bank, a multiply-accumulate (MAC) circuit, and a register file configured to store input data and an operation result. The weight handler circuit is configured to adjust a distribution of the weights based on a specification of the MAC circuit, and the MAC circuit is configured to perform an operation on the input data received from the register file and the weights based on a result of the distribution of the weights, to generate the operation result.