In-Switch Embedding Bag Pooling for Memory Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The large number of embedding tables and their sizes in neural network-based recommendation systems lead to increased memory bandwidth, reducing system performance and increasing power consumption.
Innovation Solution
A computing system with multiple processing nodes and a switch that performs reduction operations on embedding rows, reducing memory bandwidth requirements and power consumption by offloading these operations from processors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the number of embedding tables and their sizes increase to support more features, users, and content, then the neural network can handle more complex tasks, but the memory bandwidth requirement greatly increases
Solution Approach 1:
The embedding table is divided into multiple smaller tables distributed across different memory devices. Instead of having one large embedding table that requires high memory bandwidth, the data is segmented into smaller chunks that can be fetched more efficiently, reducing the bandwidth requirement while maintaining the ability to handle numerous features and users
Solution Approach 2:
The patent introduces a new dimension by storing embedding tables in a hierarchical memory structure with multiple levels (e.g., L1 cache, L2 cache, main memory). This dimensional approach allows the system to access embedding data through multiple memory hierarchies rather than a single linear memory access path, effectively reducing the perceived memory bandwidth requirement
2Adaptability or versatility
If the number of embedding tables and their sizes increase, then the neural network can support more features and content, but system performance reduces due to increased memory bandwidth requirements
Solution Approach 1:
The system performs preliminary actions by pre-loading frequently accessed embedding tables into faster memory hierarchies before they are needed during inference. This allows the neural network to maintain high performance even with a large number of embedding tables, as the most critical data is already positioned in optimal memory locations
Solution Approach 2:
The patent introduces an intermediary memory management layer that acts as a mediator between the neural network processors and the embedding tables. This intermediary layer intelligently manages memory access patterns, caches frequently used embeddings, and orchestrates data fetching operations to minimize performance degradation from the large number of embedding tables
3Quantity of substance
If embedding tables are stored in system memory rather than on-die caches, then more embedding data can be available, but memory bandwidth consumption increases significantly
Solution Approach 1:
The patent applies local quality by differentiating between frequently accessed and rarely accessed embedding data. Frequently accessed embeddings are stored in faster, closer memory (on-die caches or L1/L2 cache), while less frequently accessed embeddings are stored in slower system memory. This localized storage strategy ensures that the majority of embedding data needed for computation is available with minimal memory bandwidth consumption
Solution Approach 2:
The system creates copies of embedding tables in multiple memory locations (caches, buffers, and potentially multiple memory devices). These copies allow the neural network to access embedding data from the nearest available memory rather than always accessing system memory, reducing the effective memory bandwidth requirement while maintaining access to the full quantity of embedding data
4Adaptability or versatility
If irregular memory access operations are used to access embedding tables, then the neural network can flexibly retrieve needed data, but spatial data locality and temporal data locality cannot be utilized
Solution Approach 1:
The patent implements feedback mechanisms that monitor memory access patterns and use this information to optimize future access operations. By analyzing which embedding tables are accessed most frequently and how they are accessed, the system can adjust its memory retrieval strategies to better exploit spatial and temporal locality, thereby improving memory access efficiency while maintaining the flexibility needed for diverse neural network operations
Data Source
AI summary
An apparatus and method for reducing the memory bandwidth of executing machine learning models. A computing system includes two or more processing nodes, each including at least one or more processors and a corresponding local memory. Switch circuitry communicates with at least the local memories and a system memory of the computing system. The switch includes multiple direct memory access (DMA) interfaces. Each of one or more processing nodes stores multiple embedding rows of embedding tables. A processor of the processing node identifies two or more embedding rows as source operands of a reduction operation. The switch executes memory access requests to retrieve data of the two or more embedding rows from the corresponding local memory, and generates a result by performing the reduction operation. The switch sends the result to the local memory.