Distributed Neural Network Inferencing via Contiguous Data Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently training and operating neural networks due to high memory and time requirements, and the inability to effectively distribute computer program components and elements across multiple computer systems.
Innovation Solution
The solution involves partitioning neural network training data and activations and distributing them across multiple processors in a distributed system, allowing for parallel training of neural networks. This approach includes using a processor to cause inferencing of contiguous portions of information to be distributed between processing cores based on the location of these portions within the information relative to terminating portions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network training is performed using traditional centralized methods, then computational operations can be executed on a single system, but memory requirements and time consumption become excessively high
Solution Approach 1:
The patent divides the neural network training process into multiple segments that can be executed in parallel across different processors. Training data is partitioned into batches, and computational operations are split into forward propagation and backward propagation phases, allowing simultaneous execution on multiple processing units and reducing per-processor memory requirements.
Solution Approach 2:
The patent transitions from single-system sequential processing to multi-processor parallel processing by adding the dimension of distributed computation. Multiple processors work simultaneously on different portions of the training data or different computational stages, effectively increasing the computational throughput without proportionally increasing individual processor memory requirements.
2Loss of time
If computational operations are distributed across multiple processors, then memory and time requirements per processor are reduced, but the complexity of distributing and coordinating program components increases
Solution Approach 1:
The patent ensures continuous utilization of all processors throughout the training process by maintaining parallel execution of computational operations. Data is continuously fed to multiple processors in batches, and the system coordinates parameter updates and gradient computations without idle processing time, maximizing the efficiency of the distributed architecture.
Solution Approach 2:
The patent introduces intermediary components such as parameter servers or centralized coordination mechanisms that manage data distribution and result aggregation between workers. These intermediaries handle the complexity of coordination, allowing individual processors to focus on their computational tasks while the intermediary manages the overall training workflow and synchronizes updates.
3Ease of operation
If input data is evenly divided among processors, then workload distribution is simplified, but self-attention calculations become unbalanced due to positional dependencies in the data
Solution Approach 1:
The patent applies asymmetric distribution strategies for self-attention calculations where processors are assigned different portions of the attention matrix based on their computational capacity and the positional requirements of the data. This allows the system to maintain balanced workloads while respecting the positional dependencies inherent in self-attention mechanisms, rather than using simple even division.
Data Source
AI summary
Apparatuses, systems, and techniques to perform neural networks. In at least one embodiment, a processor is to cause information to be distributed to processing cores. In at least one embodiment, a processor is to cause inferencing of two or more contiguous portions of information to be distributed between two or more respective processing cores based, at least in part, on locations of the two or more contiguous portions within the information relative to one or more terminating portions of the information.