Two-Set Tensor Parallel Topology for Lower Communication Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for processing large tensors in deep learning models face challenges in computational efficiency and memory bottlenecks due to the difficulty in distributing tensor operations across multiple processing devices effectively.
Innovation Solution
A tensor parallel group is introduced, comprising multiple processing devices connected through direct communication channels between different sets, allowing each device to communicate with others through at most one other device, and employing techniques like tensor splitting and data reduction processes to perform operations efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tensor operations are distributed across multiple processing devices, then computational efficiency is improved, but communication complexity and data sharing requirements increase
Solution Approach 1:
The patent divides processing devices into two distinct sets (first set and second set) with different communication connectivity patterns. Processing devices in the first set directly communicate with devices in the second set, while devices in the same set communicate through at most one intermediary device. This segmentation reduces communication complexity by creating a structured topology that limits the number of direct communication channels needed.
Solution Approach 2:
The patent introduces a dimensional structure to the communication topology by organizing devices into two sets with specific communication rules. Instead of a flat fully-connected topology where every device communicates with every other device, the patent creates a two-dimensional communication structure where communication patterns depend on set membership, reducing the communication dimensionality and complexity.
2Quantity of substance
If tensor operations are distributed across multiple processing devices, then memory bottlenecks are reduced, but data sharing requirements increase
Solution Approach 1:
The patent segments tensors into partitioned representations that are distributed across processing devices in a structured manner. By dividing the tensor data according to the two-set device topology, the patent reduces the amount of data that needs to be shared between devices while maintaining the ability to perform complete tensor operations through coordinated computation across the distributed system.
Data Source
AI summary
A tensor parallel group including multiple processing devices separated into a first set of two or more of the processing devices and a second set of two or more of the processing devices. The tensor parallel group may also include multiple communication channels to directly communicatively couple every processing device in the first set of the processing devices with every processing device in the second set of the processing devices without communicatively coupling any of the processing devices in the same set of the processing devices. In these and other embodiments, the processing devices may be configured such that each of the processing devices may be able to communicate with any of the other of the processing devices through at most one other of the processing devices.