Compressed Lookup Data Exchange for Model-Parallel Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In electronic devices using model parallelism for processing input data, the significant computational and communication efforts are absorbed by acquiring and communicating lookup data between nodes, leading to increased latency and processing capacity utilization.
Innovation Solution
The implementation of compressing lookup data by identifying and removing duplicate indices from input index vectors, generating compressed sets of input index vectors, and communicating these for lookups, which reduces the amount of data exchanged between nodes, thereby reducing the number of lookups and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If model parallelism is used to distribute model data across multiple nodes, then memory capacity requirements per node are reduced, but communication overhead and latency increase due to all-to-all lookup data exchange
Solution Approach 1:
The patent merges duplicate lookup requests across multiple nodes by identifying that the same embedding table indices are frequently requested by different nodes. Instead of each node independently requesting lookup data, the system consolidates these requests into a single lookup operation, combining the results and distributing them to all requesting nodes. This eliminates redundant communication operations and reduces overall latency.
Solution Approach 2:
The patent performs preliminary identification of duplicate lookup requests before the actual data communication occurs. By analyzing the pattern of lookup requests across nodes and pre-identifying duplicates, the system prepares a consolidated set of unique lookup operations. This preliminary action prevents unnecessary communication overhead from occurring in the first place.
2Reliability
If full lookup data is communicated between all nodes, then each node has access to complete model data, but communication fabric load and processing capacity utilization increase
Solution Approach 1:
The patent extracts and removes duplicate lookup requests from the communication process. By identifying which lookup requests are duplicates across different nodes, the system extracts only the unique lookup operations that need to be performed. This extraction process eliminates redundant data transmission while ensuring all nodes still receive the necessary lookup data through the consolidated results.
3Ease of operation
If duplicate lookup requests are performed across nodes, then each node independently accesses embedding tables, but the number of lookup operations and communication overhead increase
Solution Approach 1:
The patent introduces an intermediary coordination mechanism that manages lookup requests across nodes. Instead of nodes operating completely independently or requiring complex peer-to-peer coordination, the system uses a centralized coordination approach where lookup requests are collected, deduplicated, and executed in a coordinated manner. This intermediary layer simplifies the coordination complexity while maintaining the benefit of distributed model parallelism.
Data Source
AI summary
An electronic device includes multiple nodes. Each node generates compressed lookup data to be used for processing instances of input data through a model using input index vectors from a compressed set of input index vectors for each part among multiple parts of a respective set of input index vectors. Each node then communicates compressed lookup data for a respective part to each other node.