Pipelined Compute-in-Memory Architecture for Low-Latency Weight Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing learning networks face challenges with computationally expensive weight loading and data movement, which disrupt pipelining and increase latency and power consumption, particularly in edge devices.
Innovation Solution
Implementing a pipelined compute-in-memory architecture with dedicated weight memory (W-memory) directly coupled to compute engines (CEs), using SRAM bit cells for weight storage and incorporating CIM hardware modules for parallel vector-matrix multiplications, along with local update modules for efficient weight updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If weight data is loaded from external memory to compute engines, then the learning network can perform computations, but the weight loading process becomes computationally expensive and increases latency
Solution Approach 1:
The system divides weight storage into two segments: external memory for long-term weight storage and on-chip weight memory (register files) for frequently accessed weights. This segmentation allows weights to be loaded once into on-chip memory and reused multiple times, reducing repeated loading latency and improving computation throughput.
Solution Approach 2:
Weight data is pre-loaded from external memory into on-chip weight memory before computation begins. This preliminary action ensures that weights are readily available in fast on-chip memory during the computation phase, eliminating loading latency during actual inference or training operations.
2Adaptability or versatility
If weight data is frequently loaded and stored from external memory, then the learning network can be reconfigured, but power consumption increases
Solution Approach 1:
The memory system is segmented into external memory for persistent weight storage and on-chip weight memory for active computation. This allows the system to maintain reconfigurability by loading different weight sets into on-chip memory as needed, while minimizing power consumption by keeping frequently used weights locally and reducing expensive external memory access operations.
Solution Approach 2:
The system implements local weight memory (register files) at each compute engine, allowing weights to be stored and accessed locally without repeated external memory access. This local quality enables fast reconfiguration when needed while minimizing overall power consumption during steady-state operation.
3Productivity
If data is moved between memory and compute engines, then computations can be performed, but data movement becomes computationally expensive and disrupts pipelining
Solution Approach 1:
The system merges memory and compute functions by integrating weight memory (register files) directly within compute engines. This merging eliminates the need for separate data movement between external memory and compute engines, allowing weights to be accessed directly at the computation location, thus maintaining high computation speed without data movement latency disruptions to the pipeline.
4Quantity of substance
If external memory is used for weight storage, then large weight datasets can be stored, but access latency and power consumption increase
Solution Approach 1:
Weight storage is segmented into external memory for bulk capacity and on-chip weight memory for fast access. This segmentation allows the system to store large weight datasets externally while keeping actively used weights in fast on-chip memory, achieving both large storage capacity and low access latency simultaneously.
Solution Approach 2:
On-chip weight memory (register files) acts as an intermediary between external memory and compute engines. This intermediary buffer stores weights locally, reducing the frequency of external memory access and thereby lowering access latency and power consumption while maintaining the ability to store large weight datasets externally.
Data Source
AI summary
A compute tile is described. The compute tile includes at least one general-purpose processor, a compute tile memory, compute engines, and weight memories corresponding to the compute engines. Each of the compute engines includes a compute-in-memory (CIM) hardware module. The CIM hardware module include storage cells and compute logic coupled with the storage cells. Each of the compute engines is configured to perform a vector-matrix multiplication of an input vector and weights stored in at least one of the storage cells or at least one of the weight memories.