Modular HBM Chiplet Architecture for Scalable AI Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing HBM architectures face limitations in scalability and bandwidth expansion due to physical constraints and packaging limitations, which hinder the efficient utilization of computational capabilities in AI accelerators.
Innovation Solution
A modular HBM design utilizing daisy-chain and network-grid configurations to interconnect multiple HBM chiplets, allowing scalable memory bandwidth and capacity expansion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional point-to-point HBM connections are used, then memory interface simplicity is maintained, but scalability and bandwidth expansion are limited by shoreline width and routing layers
Solution Approach 1:
The HBM system is segmented into multiple independent HBM dies, each capable of autonomous operation. Instead of a monolithic memory interface, the system divides memory capacity and bandwidth across multiple discrete dies that can be independently configured, allowing scalability without proportionally increasing interface complexity.
Solution Approach 2:
The patent transitions from traditional 2D point-to-point connections to a 3D stacked architecture with vertical interconnects. By adding the vertical dimension through multi-layer routing and inter-die connections, the system achieves higher bandwidth and scalability while maintaining a manageable 2D shoreline interface footprint.
2Productivity
If more HBM dies are stacked to increase memory bandwidth, then bandwidth capacity increases, but packaging limitations and routing layer constraints prevent further expansion
Solution Approach 1:
Multiple HBM dies are nested vertically in a stacked configuration, with each die containing complete memory functionality. This nested architecture allows high bandwidth capacity within a compact footprint, as memory capacity scales vertically rather than requiring proportional increases in lateral packaging area and routing complexity.
3Adaptability or versatility
If compute die shoreline width is increased to accommodate more memory interfaces, then more HBM dies can be connected, but compute die area and packaging size increase
Solution Approach 1:
The patent resolves the shoreline width constraint by moving memory interfaces from the 2D compute die shoreline into the 3D vertical stacking dimension. Multiple HBM dies are connected through vertical interconnects and multi-layer routing, allowing high connectivity without increasing compute die area, as the expansion occurs in the vertical rather than lateral dimension.
Data Source
AI summary
A modular high-bandwidth memory (HBM) system and method are disclosed. The system includes a compute die including a memory controller, one or more die-to-die (D2D) channels coupled to the compute die; and one or more HBM chiplets coupled to the one or more D2D channels. The one or more chiplets are configured to receive a memory access request, and process the memory access request or forward the memory access request to a subsequent HBM chiplet.