Graph-Based Recommendation Inference to Reduce Embedding Memory Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Learning Recommendation Models (DLRMs) face significant performance bottlenecks due to the high memory bandwidth requirements of sparse embedding layers, leading to increased Total Cost of Ownership (TCO) and power consumption in data centers, with existing techniques failing to effectively exploit spatial and temporal locality in memory access patterns.
Innovation Solution
A scalable graph-based algorithm system (GRACE) constructs an Item Co-occurrence Graph (ICG) to identify frequently accessed items and their combinations, clusters nodes to form cache-friendly groups, and stores partial sums in a heterogeneous memory system, optimizing cache space usage and reducing memory traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If sparse embedding layers are used in DLRM to handle large-scale item recommendations, then the model can process population-scale data, but memory bandwidth requirements increase significantly causing performance bottlenecks
Solution Approach 1:
The system performs preliminary actions by pre-computing and caching partial sums of embedding vectors for frequently co-accessed item combinations during an offline phase. This preparation allows the online inference phase to retrieve pre-computed results instead of performing full embedding lookups and reductions, significantly reducing online memory bandwidth requirements while maintaining model scalability
Solution Approach 2:
The system applies local quality by differentiating between frequently accessed and infrequently accessed item combinations. It identifies hotspots in the embedding access pattern and applies optimization (caching partial sums) specifically to these local regions of high frequency, rather than uniformly optimizing the entire embedding space, thus improving performance where it matters most
2Productivity
If existing techniques cache partial sums or exploit heterogeneous memory systems, then memory bandwidth reduction is achieved, but the approaches do not scale to large embedding tables or fail to capture global user-item interactions
Solution Approach 1:
The system employs dynamic clustering algorithms that can adapt to varying embedding table sizes and access patterns. The clustering structure is built dynamically based on actual user-item interaction data, allowing the system to scale gracefully from small to large embedding tables while maintaining optimization effectiveness. The clustering approach naturally adapts to the global structure of user-item interactions regardless of table size
Solution Approach 2:
The system transitions from flat, table-based embedding storage to a hierarchical cluster-based organization. By introducing the clustering dimension, the system creates a multi-level structure where embeddings are grouped by their co-access patterns, enabling scalable handling of large embedding tables while capturing global interaction structures that flat approaches miss
3Productivity
If MERCI stores partial sums of more than two items using expensive user trace processing, then memory traffic reduction is improved, but the algorithm does not scale to large embedding tables and operates only on sub-groups
Solution Approach 1:
The system applies partial action by selectively caching partial sums only for item combinations that exhibit high co-access frequency, rather than pre-computing and storing all possible partial sums. The clustering algorithm identifies and focuses on the most beneficial subsets, achieving significant memory traffic reduction without the excessive computational cost of processing all item combinations or requiring expensive user trace processing
Data Source
AI summary
The high memory bandwidth demand of sparse embedding layers continues to be a critical challenge in scaling the performance of recommendation models. A lightweight and scalable graph-based algorithm-system co-design framework is proposed to significantly improve the embedding layer performance of recommendation models. This framework includes a novel item co-occurrence graph that scalably records item co-occurrences. Additionally, a new system-aware graph clustering algorithm is presented to find frequently accessed item combinations of arbitrary lengths to compute and memorize their partial sums. High-frequency partial sums are stored in a software-managed cache space to reduce memory traffic and improve the throughput of computing sparse features.