Modular HBM Chiplet Architecture for Scalable Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing HBM architectures face limitations in scalability and bandwidth expansion due to physical constraints and packaging challenges, leading to a memory bottleneck that restricts the performance of AI accelerators.
Innovation Solution
A modular HBM design utilizing daisy-chain and network-grid configurations to interconnect multiple HBM chiplets, allowing scalable memory bandwidth and capacity expansion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional point-to-point HBM connections are used, then memory interface simplicity is maintained, but memory bandwidth and capacity cannot scale beyond physical constraints
Solution Approach 1:
The HBM memory system is divided into multiple independent HBM chiplets, each capable of autonomous operation. These chiplets are interconnected through D2D channels to form scalable configurations (e.g., 1-chiplet, 2-chiplet, 4-chiplet setups), allowing memory bandwidth and capacity to scale by simply adding more chiplet segments without redesigning the entire memory interface architecture.
Solution Approach 2:
The patent transitions from traditional point-to-point connections to a mesh network topology that adds spatial dimensionality to the interconnect architecture. This mesh configuration allows memory requests to be routed through multiple paths across the chiplet network, enabling bandwidth scaling beyond the limitations of linear point-to-point connections while maintaining manageable complexity through standardized routing protocols.
2Quantity of substance
If compute die shoreline width and routing layers are increased to connect more HBM dies, then memory capacity can be expanded, but manufacturing complexity and packaging difficulty increase significantly
Solution Approach 1:
Instead of creating a single large HBM die requiring extensive routing layers and shoreline width, the memory system is segmented into multiple smaller HBM chiplets. Each chiplet maintains a standardized, manageable interface that is easier to manufacture and package. The overall memory capacity is achieved by assembling these standardized chiplet segments in configurable arrangements, simplifying the manufacturing process while still providing high memory capacity.
Solution Approach 2:
The HBM chiplet design employs a universal standardized interface that can be used across different configurations and capacity requirements. This universal chiplet design allows the same basic unit to be deployed in various quantities and arrangements (1-chiplet, 2-chiplet, 4-chiplet configurations) without requiring custom manufacturing for each capacity level, significantly reducing packaging complexity and manufacturing overhead.
3Productivity
If more D2D channels are added to increase memory bandwidth, then data transfer rate improves, but the memory wall bottleneck persists due to interface limitations
Solution Approach 1:
The high-bandwidth data transfer requirement is divided across multiple independent D2D channels that connect to separate HBM chiplets. Instead of concentrating all bandwidth requirements through a single interface, the system segments the data transfer path into parallel channels, each handling a portion of the total bandwidth. This segmentation allows the system to achieve high aggregate data transfer rates while keeping individual interface complexity manageable and scalable.
Solution Approach 2:
The system implements dynamic bandwidth allocation where D2D channels can be actively configured and routed to different HBM chiplets based on workload demands. This dynamic capability allows the memory interface to adaptively scale bandwidth utilization, activating only the necessary number of channels and chiplets for each specific task, thereby maintaining high productivity while preserving interface scalability and avoiding unnecessary complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A modular high-bandwidth memory (HBM) system and method are disclosed. The system includes a compute die including a memory controller, one or more die-to-die (D2D) channels coupled to the compute die; and one or more HBM chiplets coupled to the one or more D2D channels. The one or more chiplets are configured to receive a memory access request, and process the memory access request or forward the memory access request to a subsequent HBM chiplet.