3D Compute-Memory Stack Layout for Bandwidth and Thermal Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI processing systems face limitations in I/O bandwidth and thermal management due to the stacking of dynamic random-access memory (DRAM) on top of a compute die, leading to increased latency and power consumption.
Innovation Solution
The proposed solution involves an integrated circuit package configuration where the memory die is positioned below or alongside the compute die, utilizing high-density through-silicon vias (TSVs) and micro-bumps for interconnects, allowing for direct face-to-face alignment of active devices and decoupling TSV density from micro-bump density, thereby enhancing bandwidth and thermal management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If DRAM is stacked on top of compute die, then memory bandwidth is improved, but thermal management deteriorates and latency increases
Solution Approach 1:
The patent inverts the conventional stacking order by placing the memory die below the compute die instead of above it. This inversion allows the compute die to be positioned closer to the heat sink for improved thermal management while maintaining high memory bandwidth through direct vertical interconnects. The memory die acts as a thermal barrier between the compute die and the heat sink, optimizing both performance and thermal characteristics.
Solution Approach 2:
The patent transitions from horizontal or peripheral interconnect arrangements to vertical through-silicon via (TSV) interconnects. By utilizing the vertical dimension with TSVs that penetrate through the memory die, the invention achieves high-density interconnects with shorter signal paths, improving bandwidth while enabling better thermal management through the vertical stacking architecture.
2Productivity
If DRAM is stacked on top of compute die, then memory bandwidth is improved, but latency increases
Solution Approach 1:
By inverting the stack order with memory below compute die, the patent achieves direct vertical access paths through TSVs that reduce signal propagation distance. This inversion eliminates the need for longer horizontal interconnect paths, thereby reducing latency while maintaining high bandwidth through the direct vertical connection architecture.
3Productivity
If high-density TSVs are used for interconnects, then bandwidth is improved, but manufacturing complexity increases
Solution Approach 1:
The patent segments the interconnect function into two distinct components: TSVs for vertical power and signal transport through the memory die, and micro-bumps for horizontal die-to-die bonding. This segmentation allows TSV density to be decoupled from micro-bump density, enabling high bandwidth through dense TSVs while managing manufacturing complexity by separating the functions of vertical and horizontal interconnection.
Solution Approach 2:
The patent changes the density parameters independently for TSVs and micro-bumps. By allowing different density levels for vertical (TSV) and horizontal (micro-bump) interconnects, the invention optimizes bandwidth through high TSV density while managing manufacturing complexity through appropriate micro-bump density, rather than requiring both to be uniformly high.
4Ease of manufacture
If micro-bumps are used for die coupling, then ease of manufacture is improved, but I/O bandwidth is limited
Solution Approach 1:
The patent segments the interconnect function into TSVs for high-density vertical signaling and micro-bumps for mechanical bonding and horizontal connection. This segmentation allows TSVs to carry the majority of I/O bandwidth traffic vertically through the memory die, while micro-bumps provide reliable die coupling, thereby achieving both high bandwidth and ease of manufacture.
Solution Approach 2:
The patent introduces TSVs as intermediary vertical interconnects that mediate between the micro-bump die coupling interface and the active devices within the memory die. This intermediary TSV structure enables high-bandwidth vertical signaling while maintaining the simplicity of micro-bump-based die attachment, effectively bridging the gap between ease of manufacture and high I/O bandwidth.
Data Source
AI summary
Described is a packaging technology to improve performance of an AI processing system. An IC package is provided which comprises: a substrate; a first die on the substrate, and a second die stacked over the first die. The first die includes memory and the second die includes computational logic. The first die comprises DRAM having bit-cells. The memory of the first die may store input data and weight factors. The computational logic of the second die is coupled to the memory of the first die. In one example, the second die is an inference die that applies fixed weights for a trained model to an input data to generate an output. In one example, the second die is a training die that enables learning of the weights. Ultra high-bandwidth is changed by placing the first die below the second die. The two dies are wafer-to-wafer bonded or coupled via micro-bumps.