The invention discloses a fine-grained distributed training method and
system based on gradient quantization sparse compression, proposes a dynamic
tensor fusion technology, automatically searches an optimal
tensor fusion buffer threshold by using a binary search method, merges gradient tensors of a plurality of
layers into a larger buffer for transmission, does not need manual adjustment and optimization, and improves the training efficiency. And the optimal balance between the communication
delay and the
bandwidth utilization rate is realized. A fine-grained
pipeline scheduling mechanism for communication decoupling is provided, a gradient synchronization process is decoupled into a sparse communication task in a back propagation stage and a quantitative communication task in a
forward propagation stage, fine-grained overlapping with forward and backward calculation tasks is realized, the
utilization rate of calculation resources is maximized, and communication
delay is completely masked. In the back propagation stage, a fragmentation Top-k sparsification and AlltoAll routing strategy is adopted, transmission of a large number of zero values is avoided, and the
communication complexity of back propagation is reduced; in the
forward propagation stage, a low-bit sparse quantization and AllGather strategy is adopted, and the bandwidth
bottleneck problem caused by the fact that a traditional sparse method becomes dense back in the parameter synchronization stage is solved.