Distributed Tensor Computation Layouts for Lower Communication Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently distributing tensor computations across multiple computing devices for tasks like training large neural networks, leading to suboptimal utilization of computational resources and increased communication overhead.
Innovation Solution
A system that allows users to specify the layout of tensors across computing devices, enabling efficient distribution of tensor computations through data and model parallelism, minimizing communication and optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tensor computations are distributed across multiple computing devices, then computational capacity and resource utilization are improved, but communication overhead and system complexity increase
Solution Approach 1:
The patent introduces a compiler as an intermediary system that automatically handles the complexity of distributing tensor computations across multiple devices. The compiler translates high-level tensor operations into device-specific operations and manages data partitioning, communication patterns, and coordination between devices, thereby shielding users from the underlying system complexity while achieving efficient distributed computation
2Loss of time
If tensor computations are distributed across multiple computing devices, then computation time is reduced, but communication overhead increases
Solution Approach 1:
The patent applies segmentation by automatically partitioning tensors and computations into smaller segments that can be processed in parallel across multiple devices. The compiler analyzes tensor operations and divides them into independent segments that minimize inter-device communication requirements, allowing simultaneous execution while reducing the frequency and volume of communication overhead
Solution Approach 2:
The patent employs parameter changes by dynamically adjusting computation and communication parameters based on the specific tensor operation and device configuration. The compiler optimizes parameters such as batch size, parallelism degree, and communication frequency to achieve the best balance between computation speed and communication overhead for each specific workload
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributing tensor computations across computing devices. One of the methods includes: receiving specification data that specifies a distribution of tensor computations among a plurality of computing devices, wherein each tensor computation (i) is defined to receive, as input, one or more respective input tensors each having one or more respective input dimensions, (ii) is defined to generate, as output, one or more respective output tensors each having one or more respective output dimensions, or both, wherein the specification data specifies a respective layout for each input and output tensor that assigns each dimension of the input or output tensor to one or more of the plurality of computing devices; assigning, based on the layouts for the input and output tensors, respective device-local operations to each of the computing devices; and causing the tensor computations to be executed.