Asymmetric Data Distribution for Shared Channel AI Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In network environments where a communication channel is shared, the simultaneous access by multiple computation devices degrades communication performance during the parallel processing of AI models using data parallelism.
Innovation Solution
The method involves determining whether data parallel processing is applied, calculating computation and communication times for even data distribution, and then unevenly distributing the input data to ensure a constant difference in data sizes across devices, allowing them to access the channel sequentially.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple computation devices simultaneously access a shared communication channel during parallel AI model training, then data processing throughput is improved, but communication performance degrades due to channel contention
Solution Approach 1:
The input data is segmented and unevenly distributed across multiple computation devices based on their computation speeds. Faster devices receive smaller data portions while slower devices receive larger portions, allowing all devices to complete their computation tasks in approximately the same time and access the shared communication channel sequentially rather than simultaneously, thus resolving the channel contention problem while maintaining high data processing throughput
Solution Approach 2:
Different computation devices are assigned different data portion sizes according to their individual computation capabilities. Each device receives a locally optimized data allocation that matches its processing speed, enabling synchronized completion of computation tasks and sequential channel access without degrading overall system productivity
2Ease of manufacture
If input data is evenly distributed across multiple computation devices, then data distribution simplicity is improved, but communication bottleneck occurs in shared channel networks
Solution Approach 1:
The patent introduces asymmetric data distribution where different computation devices receive different sized data portions based on their computation speeds. This asymmetric allocation eliminates the communication bottleneck by ensuring that faster devices don't finish too quickly and wait for slower devices, thereby achieving sequential channel access and improved communication efficiency while maintaining relatively simple distribution logic
Data Source
AI summary
Disclosed herein is a method for distributed training of an AI model in a channel-sharing network environment. The method includes determining whether data parallel processing is applied, calculating a computation time and a communication time when input data is evenly distributed across multiple computation devices, and unevenly distributing the input data across the multiple computation devices based on the computation time and the communication time.