Integrated In-Memory Compute Scaling for Low-Power VMM
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing learning networks face challenges in scaling to larger sizes and higher precisions without significantly increasing power consumption, particularly in edge devices, where efficient and parallel computations are desired.
Innovation Solution
The implementation of a compute engine with a compute-in-memory (CIM) hardware module that includes an array of storage cells and compute logic, capable of performing vector-matrix multiplications efficiently, along with local update modules for weight updates, reduces power consumption and improves training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the AI accelerator is scaled to larger sizes and higher precisions, then the computing capability and model capacity are improved, but the power consumption increases significantly
Solution Approach 1:
The compute engine is divided into multiple compute-in-memory modules, each handling a portion of the weight matrix. This segmentation allows the system to process computations in parallel across multiple independent units, improving throughput without requiring all units to operate at full power simultaneously, thus managing power consumption while scaling computing capability.
Solution Approach 2:
The patent replaces traditional von Neumann architecture with compute-in-memory architecture, performing computations directly within the memory array rather than transferring data between separate storage and processing units. This substitution eliminates the data movement bottleneck and reduces the need for high-bandwidth memory transfers, thereby reducing power consumption while maintaining or improving computing capability.
2Productivity
If the AI accelerator is scaled to larger sizes, then the number of parameters and model capacity increase, but the hardware reconfiguration complexity increases
Solution Approach 1:
The weight matrix is divided into multiple smaller sub-matrices stored in separate compute-in-memory modules. Each module operates independently with its own compute logic and storage cells. This modular segmentation allows the system to scale model capacity by simply adding more modules rather than reconfiguring existing hardware, as each module can be added in a standardized manner.
Solution Approach 2:
Each compute-in-memory module is designed as a universal building block that can handle different portions of weight matrices regardless of the specific model architecture or size. The modules use the same basic structure of storage cells and compute logic, allowing them to be reused and combined in various configurations to support different model capacities without requiring hardware reconfiguration.
3Productivity
If traditional von Neumann architecture is used, then the data movement between storage and processing units is required, but the computation time and power consumption increase
Solution Approach 1:
The patent merges the storage function and computation function into a single integrated compute-in-memory module. The weight matrix is stored directly within the memory array, and compute logic is integrated within the same module. This merging eliminates the need to transfer data between separate storage and processing units, performing computations in-place and significantly reducing computation time while lowering power consumption.
Solution Approach 2:
The patent replaces the traditional von Neumann architecture with a compute-in-memory architecture where computations occur directly within the memory array. This substitution eliminates the data movement bottleneck that plagues traditional architectures, as weights and activations are processed together within the same physical location, thereby reducing both computation time and power consumption.
Data Source
AI summary
A compute engine including an input buffer and a compute-in-memory (CIM) hardware module is described. The input buffer is coupled to the CIM hardware module and provides an input vector to the CIM hardware module. The CIM hardware module includes an array of storage cells and compute logic. The array of storage cells is configured to store weights corresponding to a matrix. The compute logic is configured to perform a vector-matrix multiplication (VMM) for the matrix and the input vector. The array of storage cells includes storage blocks. Each storage block includes rows and a particular number of columns corresponding to a portion of the matrix. The compute logic includes compute logic blocks. Each compute logic block corresponds to a storage block of the storage blocks. The compute logic block performs a portion of the VMM for the portion of the matrix.