Hybrid HBM-NVM Cache Controller for Memory Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face inefficiencies in training deep neural networks due to limited memory bandwidth and capacity, leading to slow training times and increased data movement overheads when using GPUs, especially with complex algorithms requiring massive datasets.
Innovation Solution
A hybrid cache memory architecture integrating high bandwidth memory (HBM) with high capacity non-volatile memory dies, managed by a cache controller for efficient data migration, allowing for concurrent high bandwidth and high capacity storage without the need for multiple GPUs or CPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If high bandwidth memory (HBM) is used to store host data, then data access bandwidth is improved, but storage capacity is limited
Solution Approach 1:
The patent merges HBM and NVM into a single hybrid cache memory package, combining the high bandwidth of HBM with the high capacity of NVM. The cache controller manages both memory types as an integrated system, allowing data to be stored in HBM for fast access while maintaining a copy in NVM for persistent storage and capacity extension.
Solution Approach 2:
The cache controller acts as an intermediary between the host processor and the memory subsystem. It manages data transfer between HBM and NVM, coordinating write operations to ensure data is first stored in HBM for immediate access, then copied to NVM for capacity preservation, thereby resolving the bandwidth-capacity tradeoff.
2Productivity
If distributed GPUs are used to train networks, then computational capacity is improved, but data movement overheads increase
Solution Approach 1:
The patent segments the memory system into HBM for active data storage and NVM for persistent storage, allowing a single GPU to access both memory types through the cache controller. This eliminates the need for distributed GPU configurations and their associated data movement overheads while maintaining high computational capacity.
3Quantity of substance
If more HBM dies are stacked to increase capacity, then storage capacity is improved, but package complexity and power consumption increase
Solution Approach 1:
Instead of stacking multiple HBM dies to increase capacity, the patent combines HBM with NVM in a single package. The HBM provides high bandwidth for active data, while NVM provides high capacity for persistent storage. This approach achieves the desired capacity without the complexity and power consumption of stacking multiple HBM dies.
4Reliability
If data is copied to NVM concurrently with HBM storage, then data retention is improved, but write operation time increases
Solution Approach 1:
The cache controller performs preliminary actions by first storing data in HBM to ensure immediate availability and correctness, then concurrently copies the data to NVM for persistent retention. This preliminary storage in HBM ensures data reliability while the concurrent copy operation minimizes the time penalty.
Solution Approach 2:
The cache controller maintains continuity of useful action by performing the HBM storage and NVM copy operations concurrently. The write operation to HBM and the copy operation to NVM proceed in parallel, ensuring that data retention is improved without significantly increasing the perceived write operation time from the host processor's perspective.
Data Source
AI summary
According to some embodiments of the present invention, there is provided a hybrid cache memory for a processing device having a host processor, the hybrid cache memory comprising: a high bandwidth memory (HBM) configured to store host data; a non-volatile memory (NVM) physically integrated with the HBM in a same package and configured to store a copy of the host data at the HBM; and a cache controller configured to be in bi-directional communication with the host processor, and to manage data transfer between the HBM and NVM and, in response to a command received from the host processor, to manage data transfer between the hybrid cache memory and the host processor.