Hyper-square interconnect topology for distributed deep learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional interconnect topologies like torus networks face issues with scalability and communication delays due to long wiring, which hinder efficient data transfer and task allocation across distributed computing nodes during neural network training.
Innovation Solution
The implementation of a hyper-square interconnect topology and advanced ring-based AllReduce operations, allowing for efficient data packet forwarding and task allocation across computing nodes without long wires, enabling scalable and efficient communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If torus topology is used for interconnect, then connectivity between computing nodes is achieved, but communication delays increase due to long wiring
Solution Approach 1:
The patent transitions from traditional 2D torus topology to a 3D interconnect architecture by stacking multiple computing node layers vertically. This dimensional change allows computing nodes to communicate through multiple spatial dimensions, reducing the physical wiring length required for inter-node communication while maintaining full connectivity. The 3D arrangement enables shorter signal paths compared to planar 2D expansions.
2Adaptability or versatility
If torus topology is used for interconnect, then computing nodes can be connected, but scalability is limited due to inability to divide computing nodes for multiple tasks
Solution Approach 1:
The patent divides the computing node network into multiple independent layers, where each layer can be independently configured and allocated to different computing tasks. This segmentation allows the system to simultaneously execute multiple tasks across different layers without interference, significantly improving task allocation flexibility and scalability while maintaining manageable interconnect complexity within each layer.
3Productivity
If conventional interconnect is used, then basic communication between nodes is possible, but communication efficiency decreases due to delays
Solution Approach 1:
By implementing a 3D interconnect architecture that stacks computing nodes vertically, the patent reduces the physical distance data packets must travel between nodes. This dimensional transition from 2D to 3D spatial arrangement directly decreases communication delay and improves overall communication efficiency, as signals traverse shorter physical paths compared to traditional planar interconnects.
Data Source
AI summary
Embodiments of the present disclosure present a hyper-square interconnect topology and advanced ring-based AllReduce operations. In some embodiments, a topology is provided that is an improvement over conventional interconnect topologies by eliminating delays associated with long wirings. In some embodiments, computing nodes are divided into sub-sections to better allocate computing tasks, and the system can be optimized to divide up the computing nodes by maximizing the number of square sub-sections in the topology. In some embodiments, the system can be optimized to select square sub-sections first for each computing task. Each sub-section can comprise some computing nodes or all computing nodes in the hyper-square interconnect topology. This flexibility allows the hyper-square interconnect topology to utilize the computing nodes more efficiently by assigning appropriate numbers of computing nodes to each computing task based on the computing need of the computing task.