Ultra high-bandwidth artificial intelligence (AI) processor with DRAM under the processor
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI processor systems face challenges with limited I/O bandwidth and thermal issues due to the conventional stacking of dynamic random-access memory (DRAM) on top of a compute die, which results in periphery constraints and inefficient heat management.
Innovation Solution
The proposed solution involves a packaging technology where the memory die is positioned below or alongside the compute die, allowing for ultra-high bandwidth communication and improved thermal management by decoupling the die-to-die I/O density from the TSV density and utilizing tight micro-bump spacing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If DRAM is stacked on top of compute die, then memory capacity is improved, but I/O bandwidth is limited due to periphery constraints
Solution Approach 1:
The patent inverts the conventional stacking order by placing the compute die on top of the DRAM die instead of DRAM on top of compute die. This inversion allows the DRAM periphery to be fully utilized for high-bandwidth die-to-die I/O connections, while the compute die periphery connects to the package substrate, thereby resolving the bandwidth limitation caused by periphery constraints.
2Quantity of substance
If DRAM is stacked on top of compute die, then memory integration is improved, but thermal management deteriorates
Solution Approach 1:
By inverting the stack order with compute die on top and DRAM below, the patent enables direct attachment of heat sinks to the compute die (the primary heat source) while the DRAM die serves as a thermal management layer underneath. This resolves the thermal management issue by allowing efficient heat dissipation paths from the compute die without compromising memory integration.
3Quantity of substance
If compute die is positioned below DRAM, then memory density is improved, but perforation requirement increases
Solution Approach 1:
The patent inverts the stack order so that the compute die is on top and DRAM is below, which eliminates the need for extensive through-silicon vias (TSVs) in the compute die. The DRAM die can be properly perforated with TSVs for memory operations without compromising the compute die structure, thereby reducing device complexity while maintaining memory density.
4Ease of manufacture
If conventional stacking is used, then manufacturing is simplified, but power efficiency deteriorates
Solution Approach 1:
The inverted stack order with compute die on top and DRAM below creates shorter and more direct signal paths between memory and compute elements, reducing capacitive loading and dynamic power consumption. This configuration maintains manufacturing feasibility while significantly improving power efficiency for AI workloads by minimizing the distance data must travel between storage and processing.
Data Source
AI summary
Described is a packaging technology to improve performance of an AI processing system. An IC package is provided which comprises: a substrate; a first die on the substrate, and a second die stacked over the first die. The first die includes memory and the second die includes computational logic. The first die comprises DRAM having bit-cells. The memory of the first die may store input data and weight factors. The computational logic of the second die is coupled to the memory of the first die. In one example, the second die is an inference die that applies fixed weights for a trained model to an input data to generate an output. In one example, the second die is a training die that enables learning of the weights. Ultra high-bandwidth is changed by placing the first die below the second die. The two dies are wafer-to-wafer bonded or coupled via micro-bumps.