Arithmetic Processing Apparatus for Accurate Allreduce with Low-Bit Integers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel computing, as the number of nodes increases, the throughput decreases due to overflow issues when using low-bit integers for gradient information aggregation, leading to a deterioration in operation accuracy during Allreduce processing.
Innovation Solution
The proposed arithmetic processing apparatus employs a method where nodes are paired to calculate average values in a pipeline, using logical product and exclusive OR operations to avoid overflow, and adjusts the order of node counting to distribute the division of values by 2, ensuring accurate averaging without sacrificing throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If low-bit integers are used for gradient information to reduce data communication amount, then throughput is improved, but operation accuracy deteriorates due to overflow during Allreduce processing
Solution Approach 1:
The patent segments the Allreduce operation into multiple stages: first dividing gradient information by the number of nodes, then performing addition operations. This segmentation prevents overflow by ensuring intermediate values remain within the representable range of low-bit integers, while still achieving accurate averaging across all nodes.
Solution Approach 2:
The patent performs division by the number of nodes as a preliminary action before addition operations. By pre-scaling the gradient values, the system prevents overflow from occurring during subsequent addition operations, thereby maintaining both throughput and accuracy when using low-bit integers.
2Reliability
If gradient information is divided by the number of nodes before addition to suppress overflow, then overflow is reduced, but operation accuracy deteriorates when the number of nodes is large
Solution Approach 1:
The patent changes the operational parameters by using low-bit integers (8-bit or 16-bit) instead of traditional high-precision floating-point numbers.配合 the preliminary division operation, this parameter change enables the system to maintain accuracy while preventing overflow, as the division scales values to fit within the limited bit representation.
3Speed
If the number of nodes in parallel computer is increased, then calculation speed is improved, but throughput decreases due to overflow in Allreduce processing
Solution Approach 1:
The patent segments the aggregation operation to perform division first, then addition. This segmentation allows the system to scale to more nodes while maintaining throughput, as each node's divided contribution remains small enough to prevent overflow when summed across many nodes.
Data Source
AI summary
An arithmetic processing apparatus includes: a plurality of nodes (N nodes) capable of communicating with each other, each of the plurality of nodes including a memory and a processor, the memory being configured to store a value and an operation result, the processor being configured to execute first processing when N is a natural number of 2 or more, n is a natural number of 1 or more, and N≠2n, wherein the first processing is configured to divide by 2 a value held by a first node, the first node being any of the plurality of nodes and a last node in an order of counting, obtain one or more node pairs by pairing remaining nodes among the plurality of nodes exception for the first node, and calculate repeatedly an average value of values held by each node pair of the one or more node pairs.