Compute-in-Memory Tile Architecture for Low-Latency AI Data Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI accelerators are not optimized for specific models, leading to inefficiencies in training and deployment, and lack scalability for large vector-matrix multiplications, requiring in-situ training and re-training for different tasks.
Innovation Solution
A compute tile architecture with compute-in-memory (CIM) modules and a general-purpose processor that bypasses the data path for data movement, allowing parallel vector-matrix multiplications and local updates, enhancing efficiency and flexibility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single AI accelerator is used for multiple different models, then device versatility is improved, but processing efficiency deteriorates due to lack of optimization for specific models
Solution Approach 1:
The system segments the AI accelerator into multiple compute tiles, each capable of being independently configured for specific models. This allows different tiles to be optimized for different models simultaneously, resolving the contradiction between versatility and efficiency by enabling model-specific optimization without sacrificing multi-model support.
Solution Approach 2:
The compute tiles are designed to be dynamically reconfigurable, allowing the system to adapt its architecture based on the specific model being processed. This dynamic reconfiguration enables the same hardware to achieve optimal performance across different models, maintaining both versatility and efficiency.
2Adaptability or versatility
If data movement involves the general-purpose processor, then data routing flexibility is improved, but processing latency increases due to processor bottleneck
Solution Approach 1:
The patent introduces dedicated data movement interfaces and buffers as intermediaries between memory and compute tiles, eliminating the need for general-purpose processor involvement in data movement. These intermediaries handle data routing flexibly while operating in parallel with computation, thus reducing latency without sacrificing routing adaptability.
Solution Approach 2:
The system replaces the sequential, processor-mediated data movement mechanism with a parallel, dedicated data path mechanism. This substitution allows data to flow directly between memory and compute tiles through specialized interfaces, bypassing the processor bottleneck while maintaining routing flexibility through configurable data paths.
3Productivity
If hardware is configured for one specific model, then processing efficiency is improved, but adaptability to different models deteriorates
Solution Approach 1:
By dividing the AI accelerator into multiple independent compute tiles, the system allows each tile to be optimized for a specific model while the overall system maintains versatility through the combination of multiple tiles. This segmentation enables simultaneous specialization without sacrificing general adaptability.
Solution Approach 2:
Each compute tile is designed with universal interfaces and reconfigurable logic that allow it to serve multiple functions across different models. While individual tiles can be optimized for specific models, their fundamental design maintains compatibility with various model architectures, achieving both efficiency and adaptability.
4Productivity
If multiple vector-matrix multiplications are performed on separate hardware, then computational capacity is improved, but system complexity increases
Solution Approach 1:
The patent merges multiple compute tiles into a unified architecture that shares common resources such as memory interfaces, control logic, and data paths. This consolidation allows multiple vector-matrix multiplications to be performed in parallel while reducing overall system complexity through resource sharing and standardized interfaces.
Solution Approach 2:
The compute tiles are designed with universal, standardized interfaces and identical functional blocks that can perform the same operations. This uniformity simplifies the overall system architecture by eliminating the need for custom designs for each computational unit, reducing complexity while maintaining high computational capacity through parallel execution.
Data Source
AI summary
A compute tile is described. The compute tile includes compute engines and a general-purpose (GP processor coupled with the compute engines. Each of the compute engines includes a compute-in-memory (CIM) hardware module. The CIM hardware module is configured to store weights corresponding to a matrix and to perform a vector-matrix multiplication (VMM) for the matrix. The GP processor is configured to control the compute engines, to receive output of the VMM for the matrix from the compute engines, and to perform a nonlinear operation on the output. The compute engines are addressable by data movement initiators. Data may be moved to and/or from the compute engines in data paths that bypass the GP processor.