Allreduce Processing for Parallel Computers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for Allreduce processing in parallel computers, such as those used in deep learning, face challenges in maintaining computational accuracy and throughput due to overflow issues when dealing with large numbers of nodes, especially when using low-bit integer representations for gradient information.
Innovation Solution
The proposed solution involves a method where each node determines whether the number of nodes is a power of 2, and if so, uses a two-group average approach for Allreduce processing, grouping nodes to calculate average values using logical products and exclusive OR operations to avoid overflow and maintain accuracy, while also pipelining calculations to enhance throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If low-bit integer representation is used for gradient information, then data communication amount is reduced and throughput is improved, but calculation accuracy deteriorates due to overflow issues when dealing with large numbers of nodes
Solution Approach 1:
The patent segments the Allreduce operation into two distinct phases: a first calculation phase that performs addition of gradient information using low-bit integers, and a second calculation phase that performs right-shift operations. This segmentation allows each phase to be optimized independently, with the first phase focusing on throughput by using efficient low-bit addition, and the second phase correcting any overflow issues through proper bit manipulation, thus resolving the contradiction between throughput and accuracy.
Solution Approach 2:
The patent changes the parameter representation by introducing a specific bit-width parameter (e.g., 8-bit or 16-bit integers) for gradient information storage and processing. By carefully managing the bit-width and using right-shift operations to handle division by the number of nodes, the system maintains calculation accuracy while benefiting from the reduced memory bandwidth requirements of low-bit representations, thereby improving throughput without sacrificing precision.
2Power
If the number of nodes is increased to improve computing power, then processing capability is enhanced, but calculation accuracy deteriorates due to overflow in aggregation operations
Solution Approach 1:
The patent segments the aggregation operation into two distinct calculation phases. The first phase accumulates gradient information from multiple nodes using low-bit integer addition, and the second phase performs right-shift operations to divide by the number of nodes. This segmentation prevents overflow by handling the division operation separately, allowing the system to scale to a large number of nodes while maintaining calculation accuracy through proper bit manipulation in each phase.
Solution Approach 2:
The patent introduces an intermediary right-shift operation as a mediator between the addition phase and the final result. This intermediary step handles the division by the number of nodes in a controlled manner, preventing overflow that would occur if all additions were performed first. The right-shift operation acts as a buffer that manages the accumulation of values from multiple nodes, enabling scalable parallel computing while preserving accuracy.
3Device complexity
If traditional Allreduce processing is used, then simplicity is maintained, but throughput deteriorates due to inefficiency in low-bit integer operations
Solution Approach 1:
The patent changes the processing parameters by using low-bit integer representations (e.g., 8-bit or 16-bit) for gradient information instead of traditional high-precision floating-point formats. This parameter change reduces the memory bandwidth requirements and increases arithmetic operation speed. The two-phase calculation approach (addition followed by right-shift) is specifically optimized for low-bit integers, maintaining simplicity in implementation while dramatically improving throughput by leveraging the efficiency of low-bit operations in modern hardware.
Data Source
AI summary
An information processing system includes: a first information processing device that transmits first data to be retained via a communication path; a second information processing device that calculates first calculation result data that is a calculation result of the first data received via the communication path and second data to be retained and transmits the calculated first calculation result data via the communication path; a third information processing device that transmits third data to be retained via the communication path; and a fourth information processing device that calculates second calculation result data that is a calculation result of the third data received via the communication path and fourth data to be retained and calculates third calculation result data that is a calculation result of the first calculation result data received via the communication path and the calculated second calculation result data.