Embedding Table Partitioning Across Heterogeneous Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing recommendation models face performance bottlenecks due to large embedding tables that are randomly accessed, leading to slow memory access and increased latency, especially in heterogeneous memory architectures, as current methods fail to optimize partitioning and deployment based on dynamic input access patterns.
Innovation Solution
A pre-optimizer and optimizer framework dynamically partitions embedding tables across heterogeneous memory architectures, utilizing a first and second level memory in FPGAs and CPUs, to minimize cache misses and latency by leveraging past access patterns and memory access capabilities, employing a learned index technique for efficient lookup.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If embedding tables are stored in heterogeneous memory architecture with static placement, then memory capacity is sufficient to hold large embedding tables, but access latency increases due to random access patterns and cache misses
Solution Approach 1:
The embedding tables are segmented and partitioned across different levels of heterogeneous memory architecture (L1 cache, L2 cache, main memory). Frequently accessed embedding tables are placed in faster L1 cache, while less frequently accessed tables are placed in L2 cache or main memory, creating a hierarchical storage structure that balances capacity and access speed.
Solution Approach 2:
The system performs preliminary analysis of access patterns during a warm-up phase to identify frequently accessed embedding tables before production workloads begin. This pre-optimization allows the system to pre-position hot embedding tables in faster memory locations, reducing initial access latency when the workload starts.
Solution Approach 3:
The memory placement of embedding tables is dynamically adjusted based on runtime access patterns. The system continuously monitors which embedding tables are accessed most frequently and repositions them in faster memory layers, while moving less frequently accessed tables to slower but larger capacity memory, making the memory hierarchy adaptive to changing workload characteristics.
2Reliability
If all embedding tables are accessed for every inference, then complete information is available for recommendation, but inference time is bounded by the slowest access and overall performance decreases
Solution Approach 1:
The system performs partial embedding table access by identifying and retrieving only the frequently accessed embedding tables that are most relevant for the current inference workload. By accessing a subset of embedding tables that covers the majority of access patterns (following Pareto principle), the system achieves sufficient recommendation quality while significantly reducing inference time compared to accessing all embedding tables.
Solution Approach 2:
The system replaces the mechanical approach of sequentially accessing all embedding tables with a smart selective access mechanism. Using access pattern analysis and predictive algorithms, the system substitutes brute-force full-table access with targeted retrieval of only necessary embedding tables, dramatically improving inference throughput while maintaining recommendation accuracy.
3Reliability
If embedding tables are frequently updated to reflect changing user behavior, then recommendation quality improves, but system complexity and update overhead increase
Solution Approach 1:
The system updates embedding tables periodically based on accumulated access pattern data rather than continuously. During periodic update cycles, the system reanalyzes access patterns and repositions embedding tables in the memory hierarchy accordingly. This periodic update approach maintains recommendation quality by adapting to changing user behavior while avoiding the complexity and overhead of continuous real-time updates.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
High-performance deployment of DNN recommendation models heavily rely on embedding tables, and their performance bottleneck lies in the latency of embedding access. To optimize the deployment of RMs, the method and system is disclosed, which leverages heterogeneous memory types on FPGAs to improve the overall performance by maximizing the availability of frequently accessed data in faster memory. The system, using a optimizer dynamically allocates table partitions of the embedding tables based on history of input access history. A pre-optimizer block disclosed determines whether smaller tables should be partitioned or placed entirely in smaller memories, improving overall efficiency. The performance of RM is improved with improvement in average embedding fetch latency and effectively inference latency via modified Round Trip computation.