Vertically Integrated HBM with Non-Volatile Memory Extension
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional High Bandwidth Memory (HBM) devices suffer from limited storage capacity and power dependency of volatile memory, which bottlenecks data transfer and processing operations, especially in high-performance computing applications like AI/ML, due to the low bandwidth communication channel with external storage devices.
Innovation Solution
Integration of both volatile and non-volatile memory dies within the HBM device using through-silicon vias (TSVs) to establish high-bandwidth communication paths, allowing for efficient data transfer and storage, with non-volatile memory acting as a memory extension to store data persistently and restore it quickly during power cycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If volatile memory is used in HBM devices, then processing speed is improved, but power consumption increases and data loss occurs during power cycles
Solution Approach 1:
The patent combines volatile memory (for high-speed processing) and non-volatile memory (for power efficiency and data persistence) into a unified HBM device structure. The volatile memory dies provide fast access for active processing, while the non-volatile memory extension provides persistent storage, merging the advantages of both memory types into a single system that operates as one logical memory space.
Solution Approach 2:
The HBM device is designed to serve multiple functions simultaneously: it acts as both high-speed volatile memory for active processing and as persistent non-volatile storage for data retention. The system can dynamically utilize either memory type based on operational needs, providing universal memory functionality that adapts to different power states and processing requirements.
2Quantity of substance
If external storage devices are used for data persistence, then storage capacity is improved, but bandwidth and data transfer speed deteriorate
Solution Approach 1:
The non-volatile memory is nested within the HBM device structure itself, integrated at the same packaging level as the volatile memory dies. This nested configuration allows both memory types to access the same high-bandwidth TSV interconnect structure, enabling the non-volatile memory to achieve transfer speeds comparable to volatile memory while providing persistent storage capacity.
Solution Approach 2:
The patent transitions from traditional external storage connectivity (using low-bandwidth package pins) to internal vertical integration using high-bandwidth TSVs. By moving the non-volatile memory from an external dimension to an internal stacked dimension within the HBM device, the system achieves both high capacity and high transfer speeds through the same vertical interconnect pathway.
3Speed
If volatile memory is used, then processing speed is improved, but data loss occurs during power down cycles
Solution Approach 1:
The system performs preliminary data transfer from volatile memory to non-volatile memory extension before power down occurs. The controller monitors power state transitions and proactively migrates critical data to the non-volatile portion, ensuring data persistence is established in advance before the power cycle that would otherwise cause data loss in volatile memory.
Solution Approach 2:
The non-volatile memory extension acts as an intermediary between the volatile memory and permanent external storage. It serves as a buffer that captures data from volatile memory during normal operation and maintains it during power cycles, mediating the transition between volatile fast storage and persistent storage without requiring external device communication.
4Device complexity
If conventional HBM architecture is used, then device complexity is reduced, but storage capacity and functionality are limited
Solution Approach 1:
The HBM device is segmented into distinct functional regions: volatile memory dies for high-speed processing, non-volatile memory extension for persistent storage, and controller logic for managing data migration. This segmentation allows each component to be optimized for its specific function while maintaining a unified interface, adding versatility without proportionally increasing overall system complexity.
Solution Approach 2:
The unified HBM device provides universal memory functionality that adapts to different operational modes. The same physical device can operate as high-speed volatile memory during active processing, switch to persistent storage mode during power cycles, and dynamically allocate space between volatile and non-volatile regions based on workload requirements, greatly enhancing adaptability within a single device architecture.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
System-in-packages (SiPs) having combined high bandwidth memory (HBM) devices, and associated systems and methods, are disclosed herein. In some embodiments, the SiP includes a base substrate (e.g., a silicon interposer), a processing unit carried by the base substrate, and a HBM device carried by the base substrate. The combined HBM device can be electrically coupled to the processing unit through one or more traces. Further, the combined HBM device can include an interface die, one or more volatile memory dies carried by the interface die (e.g., a volatile, main memory component), and one or more non-volatile memory dies carried by the one or more memory dies. The combined HBM device can also include a shared bus that is electrically coupled to the interface die, the volatile memory dies, and the non-volatile memory dies to establish communication paths therebetween.