3D AI Chip Layout With Split Memory and Compute Dies
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial intelligence (AI) processor systems face challenges in reducing latency and power consumption during model training and inference, particularly due to the hardware-intensive nature of these processes.
Innovation Solution
The segregation of matrix multiplication processes into separate memory and compute dies, utilizing high-bandwidth ferroelectric random-access memory (Fe-RAM) and matrix multiplication units (MMUs) to optimize data storage and computation, allowing for improved performance and reduced energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If matrix multiplication is performed using conventional memory and compute architectures, then processing capability is provided, but latency and power consumption increase
Solution Approach 1:
The patent divides the computational system into separate memory die and compute die, with memory die containing Fe-RAM for data storage and compute die containing MMUs for matrix multiplication operations. This segmentation allows simultaneous data access and computation, reducing latency by eliminating data transfer bottlenecks between memory and compute units.
Solution Approach 2:
The patent transitions from conventional two-dimensional planar architecture to three-dimensional stacked architecture, where memory die and compute die are vertically stacked and interconnected via through-silicon vias (TSVs). This vertical stacking reduces data access distance and enables higher bandwidth communication, significantly improving matrix multiplication speed while reducing computational latency.
2Productivity
If hardware-intensive AI processing is performed, then model training and inference capability is provided, but power consumption increases
Solution Approach 1:
The patent employs Fe-RAM with non-volatile memory cells that maintain data without power, reducing refresh overhead and power consumption. The memory die uses high-density Fe-RAM technology with lower energy per access compared to conventional DRAM, while the compute die uses energy-efficient MMU architecture optimized for matrix multiplication operations, achieving lower overall power consumption while maintaining AI processing capability.
Solution Approach 2:
The patent implements persistent memory in Fe-RAM that retains data across power cycles, eliminating the need for continuous refresh operations required by volatile memory. This allows the system to maintain computational state without continuous power consumption, enabling energy-efficient AI processing while preserving model training and inference capabilities.
3Speed
If high bandwidth memory access is implemented, then data transfer speed improves, but device complexity increases
Solution Approach 1:
The patent integrates high-bandwidth memory interfaces and interconnect structures directly into the memory die and compute die packages, combining multiple functions into unified architectures. The TSV-based interconnect system merges data, address, and control signals into integrated communication paths, achieving high data transfer speeds while reducing the complexity of external interconnect requirements.
Data Source
AI summary
Matrix multiplication process is segregated between two separate dies—a memory die and a compute die to achieve low latency and high bandwidth artificial intelligence (AI) processor. The blocked matrix-multiplication scheme maps computations across multiple processor elements (PE) or matrix-multiplication units. The AI architecture for inference and training includes one or more PEs, where each PE includes memory (e.g., ferroelectric (FE) memory, FE-RAM, SRAM, DRAM, MRAM, etc.) to store weights and input/output I/O data. Each PE also includes a ring or mesh interconnect network to couple the PEs for fast access of information.