Dynamical Memory CIM Tiles for On-Chip Large-Model VMMs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI computing systems face inefficiencies in processing large numbers of parameters due to slow local memory technologies like DRAM, DDRx, eDRAM, and HBM, which limit the scalability and flexibility of compute-in-memory architectures, particularly for large language models.
Innovation Solution
Implementing a hardware accelerator tile with compute-in-memory (CIM) modules, local memory, and stationary memory units, such as SRAM, to cache weights and perform vector matrix multiplications in parallel, reducing data movement and increasing on-chip memory density.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If DRAM, DDRx, eDRAM, or HBM memory technologies are used for local memory, then larger model capacity can be supported, but processing speed and efficiency deteriorate
Solution Approach 1:
The patent segments the memory system into two distinct parts: high-speed SRAM memory for actively used parameters and high-capacity DRAM/HBM memory for model storage. This segmentation allows the system to support large models while maintaining fast processing speeds for the parameters currently in use, resolving the contradiction between model capacity and processing speed.
Solution Approach 2:
The patent introduces SRAM memory as an intermediary between the DRAM/HBM storage and the compute-in-memory modules. This intermediary layer acts as a buffer that holds frequently accessed parameters in high-speed memory, enabling fast processing while the larger model resides in high-capacity but slower memory.
2Productivity
If more parameters are processed on-chip, then computational efficiency improves, but data movement overhead increases
Solution Approach 1:
The patent merges the memory storage function with the compute function by implementing compute-in-memory modules that can directly process parameters stored in SRAM without requiring data to be moved to separate processing units. This integration eliminates data movement overhead while maintaining high computational efficiency for parameters kept in on-chip memory.
3Duration of action of moving object
If parameter swapping is avoided, then processing continuity improves, but on-chip memory requirements increase
Solution Approach 1:
The patent adds a temporal dimension to memory usage by implementing a caching strategy where frequently accessed parameters are pre-loaded into SRAM before they are needed for computation. This allows the system to maintain processing continuity without requiring all parameters to be simultaneously available in on-chip memory, effectively managing memory requirements while avoiding parameter swapping.
Data Source
AI summary
A hardware accelerator tile for performing vector matric multiplications (VMMs) using a set of parameters, and a method for loading the parameters to compute engines of the hardware accelerator tile for use in the VMMs. The hardware accelerator tile includes (i) a plurality of compute engines respectively including compute-in-memory (CIM) modules configured to perform, in parallel, VMMs on stored parameters, (ii) one or more stationary memory units coupled with the plurality of compute engines, and (iii) local memory coupled with the plurality of compute engines.