Embedding Table Sharding for Deep Learning Training Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face inefficiencies in training due to high communication costs and memory requirements when using sparse allreduce operations for large embedding tables, which hinder scalability and increase latency and memory footprints.
Innovation Solution
The approach involves treating small embedding table gradients as dense gradients and employing vertical splitting of larger tables into smaller portions, allowing for efficient communication and processing across multiple nodes using all-to-all operations, thereby reducing latency and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sparse allreduce operations are used for large embedding tables, then gradient updates can be performed, but communication costs and memory requirements increase significantly
Solution Approach 1:
The embedding table is divided into multiple sharded portions distributed across different compute nodes. Each node maintains only a subset of the embedding table entries, reducing individual memory requirements. During forward propagation, lookup entries are gathered from relevant shards, and during backward propagation, gradients are distributed to the appropriate shards, eliminating the need for full-table communication.
Solution Approach 2:
The patent introduces a new dimension of organization by sharding embedding tables across multiple nodes rather than maintaining them centrally. This dimensional transformation allows the system to handle large embedding tables by distributing them spatially across the compute cluster, reducing communication overhead through targeted gradient transmission to specific shards.
2Reliability
If sparse allreduce operations are used for large embedding tables, then gradient updates can be performed, but memory requirements increase
Solution Approach 1:
The embedding table is segmented into multiple sharded portions, with each compute node storing only a subset of entries. This segmentation reduces the memory footprint at each node while maintaining the ability to perform complete gradient updates through coordinated communication during backward propagation.
Solution Approach 2:
Each compute node maintains local copies of only the embedding table entries relevant to its assigned parameters. This local quality optimization reduces memory requirements at each node while ensuring that gradient updates can be performed efficiently by communicating only with nodes holding the relevant shards.
3Adaptability or versatility
If embedding tables are distributed across multiple nodes, then scalability is improved, but communication latency increases
Solution Approach 1:
The embedding table is segmented and distributed across multiple compute nodes, enabling the system to scale to larger models and datasets. During training, only the necessary shards are communicated between nodes, reducing overall communication latency compared to centralized approaches while maintaining scalability.
Solution Approach 2:
Embedding table shards are pre-distributed to compute nodes before training begins. This preliminary action eliminates the need for repeated full-table communications during training, reducing latency by ensuring that each node has immediate access to its assigned shards without requiring frequent synchronized updates.
Data Source
AI summary
Systems, apparatuses and methods may provide for technology that identifies an embedding table associated with a neural network. The neural network is associated with a plurality of compute nodes. The technology further identifies a number of entries of the embedding table, and determines whether to process gradients associated with the embedding table as dense gradients or sparse gradients based on the number of entries.