Interface-Die Cache in Hybrid HBM for Bandwidth and Capacity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing high-bandwidth memory (HBM) devices in system-in-package (SiP) configurations face limitations in storage capacity and power consumption due to their volatile nature, while external storage devices provide sufficient capacity but low bandwidth, leading to bottlenecks and inefficiencies in data access.
Innovation Solution
Integration of a hybrid HBM device with an additional L3 cache memory on an interface die, coupled to memory dies via a high-bandwidth bus, which reduces the need for direct access to volatile memory dies by caching frequently accessed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If external storage devices are used to provide sufficient storage capacity, then storage capacity is improved, but bandwidth deteriorates leading to bottlenecks
Solution Approach 1:
The memory system is segmented into multiple functional layers: volatile memory dies for high-speed access, non-volatile memory die for capacity expansion, and cache memory on the interface die for coordination. This segmentation allows each layer to specialize in its strength while working together to achieve both high bandwidth and large capacity
Solution Approach 2:
Cache memory is introduced as an intermediary component on the interface die between the volatile memory dies and external storage devices. This cache memory buffers data transfers, managing the bandwidth bottleneck by pre-loading and caching frequently accessed data, thereby maintaining high transfer speeds while accessing large external storage capacity
2Speed
If volatile memory is used to provide high bandwidth, then bandwidth is improved, but storage capacity deteriorates due to volatile nature
Solution Approach 1:
The system merges volatile memory (for speed) and non-volatile memory (for capacity) into a unified hybrid memory device. The volatile memory dies provide high-bandwidth access while the non-volatile memory die provides persistent storage capacity, and both are managed together through the interface die with cache memory to deliver both speed and capacity simultaneously
Solution Approach 2:
The interface die serves multiple functions: it controls the volatile memory dies, manages the non-volatile memory die, operates cache memory, and coordinates data transfers between all components. This multi-functional design allows the system to achieve both high bandwidth and large storage capacity through a single integrated device
3Speed
If direct access to volatile memory dies is increased to improve data access, then bandwidth is improved, but power consumption deteriorates
Solution Approach 1:
The cache memory on the interface die performs preliminary actions by pre-loading and caching frequently accessed data before it is needed by the processor. This allows data to be ready in the cache, reducing the need for repeated high-power accesses to the volatile memory dies and thereby reducing overall power consumption while maintaining fast data access
Data Source
AI summary
System-in-packages (SiPs) having hybrid high bandwidth memory (HBM) devices, and associated systems and methods, are disclosed herein. In some embodiments, the SiP includes a base substrate, as well as a processing device and a hybrid high-bandwidth memory (HBM) device each carried by the base substrate. The processing device includes a processing unit and a first cache memory associated with a first level of a cache hierarchy. The hybrid HBM device is electrically coupled to the processing unit through a SiP bus in the base substrate. Further, the hybrid HBM device includes an interface die, one or more memory dies carried by the interface die, and a shared bus electrically coupled to the interface die and each of the memory dies. The hybrid HBM device also includes a second cache memory formed on the interface die that is associated with a second level of the cache hierarchy.