Associative Summation for High Performance Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In High Performance Computing (HPC) systems, the summation of floating-point numbers is not associative, leading to inconsistencies and loss of precision due to the order of arrival, which increases memory overhead, bandwidth consumption, and latency, and restricts flexibility in network topology changes.
Innovation Solution
Converting floating-point numbers to a derived floating-point representation with signed integer multiplicands corresponding to different orders of magnitude, allowing for associative summation independent of the order of arrival, and then converting back to the original representation for communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If floating-point numbers are summed in standard HPC networks, then the summation result depends on the order of arrival, but implementing order-independent summation increases memory overhead and bandwidth consumption
Solution Approach 1:
The floating-point number is segmented into multiple components: sign bit, exponent, and mantissa. The mantissa is further divided into integer and fractional parts. This segmentation allows each component to be processed separately, enabling order-independent summation by aligning and summing corresponding components across different numbers without requiring additional memory storage.
Solution Approach 2:
The patent transforms the standard floating-point representation into a modified format where the mantissa is explicitly separated into integer and fractional parts. This parameter change in representation allows the summation operation to be performed component-wise, ensuring that the final sum is independent of the order in which numbers are processed, while avoiding increased memory overhead.
2Reliability
If floating-point numbers are summed in standard HPC networks, then the summation result depends on the order of arrival, but implementing order-independent summation increases latency
Solution Approach 1:
By segmenting the floating-point number into sign, exponent, and mantissa components, and further dividing the mantissa into integer and fractional parts, the patent enables parallel processing of corresponding components. This segmentation allows the summation to proceed without waiting for complete numbers to arrive in a specific order, reducing latency while maintaining consistency.
Solution Approach 2:
The summation process is designed to be self-sufficient at each network node. Each node can independently sum the integer and fractional parts of received numbers without requiring coordination or additional time for order verification, enabling order-independent summation that reduces latency.
3Reliability
If floating-point numbers are summed in standard HPC networks, then the summation result depends on the order of arrival, but implementing order-independent summation restricts flexibility in network topology changes
Solution Approach 1:
Each network node performs self-sufficient summation of floating-point numbers using the component-wise method. This self-service capability means nodes do not need to coordinate with each other or follow fixed communication patterns, allowing the network topology to be changed or reconfigured without affecting the summation consistency. The method is inherently adaptable to different network architectures.
Solution Approach 2:
The component-wise summation method is universally applicable to any network topology. Whether nodes are arranged in a mesh, tree, ring, or any other configuration, the same algorithm can be applied at each node to achieve order-independent summation, providing versatility across different HPC network architectures.
Data Source
AI summary
Apparatus, systems, and methods are described, including apparatus that includes one or more communication interfaces for communicating over a communication network, and a processor. The processor is configured to receive, via the communication interfaces, a plurality of numbers, and calculate a sum of the numbers that is independent of an order in which the numbers are received, by (i) converting any of the numbers that are received in a floating-point representation to a derived floating-point representation that includes a plurality of signed integer multiplicands corresponding to different respective orders of magnitude, and (ii) summing the numbers in the derived floating-point representation, by separately summing integer multiplicands that correspond to the same order of magnitude. Other embodiments are also described.