3D AI Processor Memory-Under-Compute Stack for Thermal Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI processing systems face limitations in I/O bandwidth and thermal management due to the stacking of dynamic random-access memory (DRAM) on top of a compute die, leading to increased latency and power consumption.
Innovation Solution
The proposed solution involves an integrated circuit package configuration where the memory die is positioned below or alongside the compute die, utilizing high-density through-silicon vias (TSVs) and micro-bumps for interconnects, allowing for ultra-high bandwidth and improved thermal management by decoupling TSV density from micro-bump density, and using various memory types such as DRAM, FeRAM, and MRAM.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If DRAM is stacked on top of compute die, then memory bandwidth is improved, but thermal management deteriorates and latency increases
Solution Approach 1:
The patent inverts the conventional stacking order by placing the memory die below the compute die instead of above it. This inversion allows the compute die to be positioned closer to the package substrate and heat sink, improving thermal management while maintaining high memory bandwidth through direct coupling via micro-bumps and TSVs.
Solution Approach 2:
The patent transitions from a two-dimensional planar arrangement to a three-dimensional stacked architecture with memory below and compute above. This vertical arrangement in another dimension enables simultaneous optimization of thermal pathways (through the substrate) and memory-access pathways (through micro-bumps and TSVs), resolving the contradiction between bandwidth and thermal management.
2Productivity
If DRAM is stacked on top of compute die, then memory bandwidth is improved, but latency increases
Solution Approach 1:
By inverting the stack order with memory below compute die, the patent shortens the signal path for memory access while improving thermal pathways. The direct coupling through micro-bumps and TSVs in this inverted configuration reduces access latency compared to conventional stacking, simultaneously achieving high bandwidth and low latency.
3Productivity
If high-density TSVs are used for interconnects, then bandwidth is improved, but manufacturing complexity increases
Solution Approach 1:
The patent segments the interconnect function into two distinct components: micro-bumps for die-to-die coupling and TSVs for through-die vertical interconnects. This segmentation allows each component to be optimized independently - micro-bumps provide high-density lateral connections while TSVs provide vertical pathways - reducing overall manufacturing complexity compared to using only high-density TSVs throughout.
Solution Approach 2:
The patent introduces micro-bumps as an intermediary element between the compute die and memory die. These micro-bumps serve as intermediate connection points that simplify the interconnect architecture by handling die-to-die coupling separately from the TSV-based vertical interconnects, thereby reducing the complexity burden on the TSV fabrication process.
4Productivity
If memory die is positioned above compute die, then I/O bandwidth is improved, but I/O bandwidth is limited by periphery constraints
Solution Approach 1:
The patent inverts the conventional arrangement by placing memory below compute die. This inversion allows the active devices of both dies to face each other, enabling direct face-to-face coupling through micro-bumps. This configuration bypasses periphery constraints by creating multiple I/O pathways through the bulk of the dies rather than being limited to edge-based connections.
Solution Approach 2:
The patent moves I/O connections from a two-dimensional periphery-based arrangement to a three-dimensional volume-based arrangement using TSVs and micro-bumps. This dimensional transition enables I/O pathways to traverse through the interior of the dies, multiplying the available bandwidth channels beyond what periphery constraints would allow in a planar configuration.
Data Source
AI summary
Described is a packaging technology to improve performance of an AI processing system. An IC package is provided which comprises: a substrate; a first die on the substrate, and a second die stacked over the first die. The first die includes memory and the second die includes computational logic. The first die comprises DRAM having bit-cells. The memory of the first die may store input data and weight factors. The computational logic of the second die is coupled to the memory of the first die. In one example, the second die is an inference die that applies fixed weights for a trained model to an input data to generate an output. In one example, the second die is a training die that enables learning of the weights. Ultra high-bandwidth is changed by placing the first die below the second die. The two dies are wafer-to-wafer bonded or coupled via micro-bumps.