Split-Die AI Processor Architecture for Low-Latency Matrix Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current artificial intelligence (AI) processor systems face challenges in reducing latency and power consumption during the training and inference processes, which are hardware-intensive and require significant processing power.
Innovation Solution
The implementation of a low latency and high bandwidth AI processor architecture that segregates matrix multiplication between two separate dies, utilizing ferroelectric memory for high-speed and large memory capacity, and employs a ring or mesh interconnect to optimize data flow and parallelize computations across multiple processing elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If matrix multiplication is performed using conventional processor architecture, then processing capability is provided, but latency is high and bandwidth is limited
Solution Approach 1:
The processor is divided into multiple processing elements (PEs) organized in a grid architecture, where each PE can independently perform matrix multiplication operations. This segmentation enables parallel processing of large matrices, significantly improving throughput and reducing latency compared to sequential processing in conventional architectures.
Solution Approach 2:
The patent transitions from conventional two-dimensional planar integration to three-dimensional vertical stacking of processing layers. Multiple processing layers are stacked vertically with inter-layer communication pathways, enabling data to flow through multiple layers simultaneously and providing exponential scaling of processing capacity without increasing footprint area.
2Quantity of substance
If conventional memory architecture is used, then data storage is provided, but memory bandwidth and speed are insufficient for AI workloads
Solution Approach 1:
The memory hierarchy is organized with multiple levels of caching memory nested within each processing element, with L1 cache directly attached to each PE, L2 cache shared among groups of PEs, and L3 cache shared across the entire processor. This nested structure enables fast access to frequently used data while maintaining large total memory capacity, resolving the bandwidth-capacity tradeoff.
3Productivity
If more processing elements are added to increase parallelism, then computation speed improves, but power consumption increases
Solution Approach 1:
The processor employs dynamic voltage and frequency scaling (DVFS) that periodically adjusts the operating frequency of processing elements based on workload demands. During low-utilization periods, frequency is reduced to minimize dynamic power consumption, while during high-utilization periods, frequency is increased to maximize throughput, achieving energy-efficient parallel processing.
4Productivity
If data is moved between memory and processor frequently, then computation is maintained, but energy is wasted on data transfer
Solution Approach 1:
The processor pre-loads data from main memory into the nested cache hierarchy before it is actually needed for computation. Data prediction algorithms anticipate future data access patterns and proactively transfer data to faster cache layers, eliminating wait states and reducing the frequency of expensive main memory accesses, thereby maintaining computation continuity while minimizing energy-wasting data transfers.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach significantly speeds up matrix multiplication operations, achieving up to 10× performance improvement and 10× lower energy consumption, while reducing latency and power usage, especially for larger matrix sizes.
Implementation Method 1
utilizing ferroelectric memory for high-speed and large memory capacity
Data Source
AI summary
Matrix multiplication process is segregated between two separate dies—a memory die and a compute die to achieve low latency and high bandwidth artificial intelligence (AI) processor. The blocked matrix-multiplication scheme maps computations across multiple processor elements (PE) or matrix-multiplication units. The AI architecture for inference and training includes one or more PEs, where each PE includes memory (e.g., ferroelectric (FE) memory, FE-RAM, SRAM, DRAM, MRAM, etc.) to store weights and input/output I/O data. Each PE also includes a ring or mesh interconnect network to couple the PEs for fast access of information.