AI Training All-Reduce Compression Over Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex neural networks require significant training time, cost, and power consumption due to increased data bandwidth needs, despite advancements in hardware and communication technologies.
Innovation Solution
The implementation of a system that uses systematic data compression and the All-Reduce algorithm over a distributed system, combined with hardware-based or software-based compression modules, to reduce interconnect communications bandwidth and power consumption, while improving training performance without precision loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If advanced servers with faster interfaces are used to improve computing and communications, then training speed is improved, but hardware cost increases
Solution Approach 1:
The patent changes the parameter of data representation by applying compression algorithms (e.g., floating-point to integer conversion, sparse representation) to transform data into a more compact form. This allows standard hardware to achieve higher effective bandwidth by processing compressed data, resolving the contradiction between training speed and hardware cost
Solution Approach 2:
The patent introduces compression and decompression modules as intermediary components between the GPU and interconnect. These intermediaries transform data into a compact form before transmission and restore it after reception, enabling efficient communication without requiring expensive high-bandwidth hardware interfaces
2Adaptability or versatility
If more data bandwidth is provided during training, then neural network complexity is improved, but power consumption increases
Solution Approach 1:
The patent applies data compression techniques that reduce the bit-width representation of gradient and parameter data (e.g., converting from 32-bit floating point to 8-bit integer or sparse formats). This parameter transformation maintains the essential information needed for complex neural network training while significantly reducing the power consumption associated with data transmission across the interconnect
3Quantity of substance
If data is compressed before transmission, then bandwidth requirement is reduced, but processing time is added
Solution Approach 1:
The patent performs compression as a preliminary action before data transmission, converting gradient and parameter data into compact representations ahead of time. This allows the compressed data to be transmitted efficiently across the interconnect, with the compression overhead being offset by the reduced transmission time and the ability to perform computations on compressed data
Data Source
AI summary
According to various embodiments, methods and systems are provided to accelerate artificial intelligence (AI) model training with advanced interconnect communication technologies and systematic zero-value compression over a distributed training system. According to an exemplary method, during each iteration of a Scatter-Reduce process performed on a cluster of processors arranged in a logical ring to train a neural network model, a processor receives a compressed data block from a prior processor in the logical ring, performs an operation on the received compressed data block and a compressed data block generated on the processor to obtain a calculated data block, and sends the calculated data block to a following processor in the logical ring. A compressed data block calculated from corresponding data blocks from the processors can be identified on each processor and distributed to each other processor and decompressed therein for use in the AI model training.