Allreduce Data Transfer Optimization in Torus Topology Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional parallel computation systems face challenges in reducing data transfer amounts during the Allreduce process, especially when using a halving and doubling method, due to uneven data communication and computation, and the differing roles of main and aggregate operation devices in torus topology systems.
Innovation Solution
The method involves aggregate operation devices acquiring array data from main operation devices, determining the order of dimensions, repeatedly halving and distributing data, and then transmitting operation results back to the main operation devices, optimizing data transfer by executing halving and doubling processes in specific dimensions to minimize data size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a traditional halving and doubling method is used in the Allreduce process, then the amount of communication data is reduced by half for each communication step, but the main operation devices are not used for computation and data is stored only in aggregate operation devices
Solution Approach 1:
The patent makes main operation devices perform dual functions: they execute main operations on input data and also participate in aggregate operations during the Allreduce process. This eliminates the need for separate aggregate operation devices, as main operation devices can both compute and aggregate data, thereby reducing system structure complexity while maintaining data reduction efficiency.
Solution Approach 2:
The patent merges the functions of main operation devices and aggregate operation devices into a single unified device. Instead of having separate devices for main operations and aggregate operations, the system combines these functions so that main operation devices perform both types of operations, simplifying the overall system architecture.
2Speed
If main operation devices are directly coupled to each other in a torus topology, then data transfer speed is improved, but the amount of data to be transferred remains large
Solution Approach 1:
The patent segments the Allreduce process into multiple communication steps, where in each step only a portion of the data is transferred between adjacent devices in the torus topology. By dividing the data transfer into incremental steps rather than transferring all data at once, the system maintains high transfer speed while reducing the amount of data transferred in each communication step.
Solution Approach 2:
The patent performs preliminary local aggregation at each main operation device before data is transferred across the network. Each device first aggregates its local data, then transfers only the aggregated results to adjacent devices in subsequent communication steps. This preliminary action reduces the volume of data that needs to be transferred across the torus topology.
3Productivity
If aggregate operation devices are used for computation in the Allreduce process, then data aggregation is efficient, but the main operation devices are not utilized and resource utilization is low
Solution Approach 1:
The patent enables main operation devices to perform both main operations and aggregate operations, making them multi-functional. This eliminates idle aggregate operation devices and ensures that all main operation devices are fully utilized throughout the computation and aggregation processes, improving overall resource utilization while maintaining aggregation efficiency.
Solution Approach 2:
The patent allows main operation devices to serve themselves by performing aggregate operations on their own data and the data received from other devices. Instead of relying on separate aggregate operation devices, each main operation device independently performs the aggregation function, maximizing resource utilization and eliminating wasted computational capacity.
Data Source
AI summary
A method for controlling an information processing system, the information processing system including multiple information processing devices coupled to each other, each of the multiple information processing devices including multiple main operation devices and multiple aggregate operation devices that are coupled to each other, the method includes: acquiring, by each of the aggregate operation devices, array data items from a main operation device coupled to the concerned aggregate operation device; determining the order of dimensions in which a process is executed and in which the information processing devices are coupled to each other; executing for each of the dimensions in accordance with the order of the dimensions, a process of halving the array data items and distributing the array data items to information processing devices arranged in the dimension; executing a process of transmitting, to information processing devices arranged in the dimension, operation results calculated based on data items.