Embedding Table Segmentation for Memory Bandwidth Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing systems face inefficiencies in memory access and data processing during embedding operations, particularly in recommendation systems that rely on deep learning, due to insufficient host memory capacity leading to bandwidth issues.
Innovation Solution
A computing system architecture that divides embedding tables into two groups, with one group stored in the host and the other in the memory system, allowing for distributed processing and efficient data management by dividing feature tables based on the number of elements, thereby optimizing memory access and reducing data movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If embedding tables are stored entirely in host memory, then processing speed is improved, but memory capacity requirements increase and bandwidth issues occur
Solution Approach 1:
The embedding table is divided into multiple segments and distributed across different storage locations (host memory and memory system). The host stores frequently accessed embedding tables while less frequently accessed tables are stored in the memory system, resolving the contradiction between processing speed and memory capacity requirements.
2Productivity
If all embedding tables are processed by the host, then processing efficiency is improved, but data movement volume increases causing bandwidth issues
Solution Approach 1:
Embedding tables that are not frequently accessed are extracted from the host memory and stored in the memory system. The host only processes embedding tables that are currently needed, significantly reducing data movement volume while maintaining processing efficiency for active tables.
3Loss of substance
If embedding tables are divided into multiple groups and distributed, then data movement is reduced, but system complexity increases
Solution Approach 1:
A management module is introduced as an intermediary to handle the complexity of distributing and managing embedding tables across multiple storage locations. This module automatically determines which embedding tables to store where based on access patterns, reducing manual system complexity while achieving reduced data movement.
Data Source
AI summary
A computing system capable of reducing data movement during an embedding operation and efficiently processing the embedding operation includes a host and a memory system. The host divides a plurality of feature tables, each including a respective plurality of embedding vectors for a respective plurality of elements, into a first feature table group and a second feature table group; generates a first embedding table configured of the first feature table group; and sends a request for a generation operation of a second embedding table configured of the second feature table group to the memory system. The memory system generates the second embedding table according to the generation operation request provided by the host. The host divides the plurality of feature tables into the first feature table group and the second feature table group based on the number of elements included in each of the plurality of feature tables.