DRAM-RRAM Hybrid Memory Mapping to Reduce DLRM PIM Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Learning Recommendation Models (DLRM) inference in data centers is bottlenecked by the high memory bandwidth requirement of the sparse embedding layer, which occupies 80% of execution time, and existing RRAM PIM solutions suffer from bit-serial latency issues, leading to performance degradation.
Innovation Solution
A hybrid memory system using DRAM and RRAM with a two-tier design where frequently accessed items are stored in RRAM for parallel processing and less frequent items in DRAM, employing Memristor Aided IoGIC (MAGIC) for parallel in-memory computation, and selectively offloading computations to the CPU to optimize performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If RRAM PIM is used for parallel in-memory computation, then memory bandwidth is improved, but bit-serial latency increases computation time
Solution Approach 1:
The patent segments the embedding table into two parts: frequently accessed items stored in RRAM for parallel processing, and less frequently accessed items stored in DRAM. This segmentation allows the system to exploit RRAM's high bandwidth for hot data while avoiding its latency issues for cold data, resolving the contradiction between bandwidth and latency.
Solution Approach 2:
The patent applies local quality by using different memory technologies for different data characteristics. RRAM is used specifically for frequently accessed embedding vectors where parallel processing provides benefit, while DRAM handles other data. This localized application of RRAM's parallel processing capability optimizes the bandwidth-latency tradeoff.
2Productivity
If frequently accessed items are stored in RRAM for parallel processing, then processing speed is improved, but system complexity increases
Solution Approach 1:
The hybrid memory system segments data storage across two memory types (RRAM and DRAM) based on access frequency. This segmentation enables parallel processing of frequently accessed items in RRAM, improving productivity while managing complexity through clear separation of concerns and specialized handling for each memory type.
3Loss of energy
If embedding vectors are stored in hybrid memory, then energy consumption is reduced, but data access complexity increases
Solution Approach 1:
The system performs preliminary action by pre-processing historical data to identify frequently accessed items and storing them in RRAM before inference. This preliminary organization of data based on access patterns reduces energy consumption during inference by enabling parallel processing, while the complexity of data access is managed through the established two-tier memory architecture.
Data Source
AI summary
The performance of modern recommendation models is limited because of the memory bandwidth-hungry embedding layer reductions. A novel hybrid memory system is presented with DRAM and RRAM with PIM capability. The performance of traditional RRAM PIM is limited by the latency of bit-serial computation. The hybrid memory system presents a comprehensive optimization approach that includes access-pattern aware mapping, compute complexity reduction, and selective PIM reduction to offset this computation latency. An evaluation shows that the hybrid memory system offers significant performance, energy, and EDP improvement of 2.6×, 1.7×, and 4.4×, on average, compared to a CPU baseline.