Tree Topology Computing System for Distributed Training Efficiency

The patent segments the distributed training system into multiple layers with different functions: parameter servers at the top layer handle aggregation, middle layers perform partial aggregation and forwarding, and bottom layers execute computing tasks. This segmentation allows aggregation operations to be distributed across multiple nodes rather than concentrated on a single node, improving both computing efficiency and aggregation speed by parallelizing the aggregation process across the hierarchical structure.

The patent introduces a hierarchical dimension to the distributed training architecture, organizing computing nodes into multiple layers. This dimensional change transforms the flat network structure into a tree-based hierarchy where data transmission occurs through multiple levels. The hierarchical structure reduces bandwidth consumption by enabling partial aggregation at intermediate layers, so that not all nodes need to communicate directly with the top layer, thereby scaling the system to handle larger data volumes without proportionally increasing bandwidth requirements.

The patent implements a nested hierarchical structure where parameter servers are nested at the top layer, middle layers contain aggregation and forwarding nodes, and bottom layers contain computing nodes. Each layer is nested within the broader system structure, with data flowing through nested levels. This nesting approach improves aggregation efficiency by allowing intermediate layers to perform partial aggregation before data reaches the top layer, reducing the total data transmission volume while maintaining a manageable network structure through modular layering.