Parallel Processing Elements Resolve Data Dependencies
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel computing methods for reentrant operations with data dependency, such as Softmax and Layer Norm layers in neural networks, face inefficiencies due to data dependencies that hinder the advantages of parallel processing by requiring data to be routed back to the same processing element, reducing the effectiveness of distributing data across multiple processing elements.
Innovation Solution
An integrated circuit system with a plurality of processing elements and a controller, where each processing element performs initial processing on data chunks, sends tokens to the controller upon completion, and receives intermediate results to resolve data dependencies, allowing for reentrant computations across multiple elements, thereby maintaining parallel processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is distributed across multiple processing elements for parallel processing, then processing speed and throughput are improved, but data dependencies require data to be routed back to the same processing element, reducing parallel processing effectiveness
Solution Approach 1:
The patent segments the computation into distinct phases: a first iteration where processing elements compute partial results independently in parallel, and a second iteration where intermediate results are aggregated and used to complete the computation. This segmentation allows parallel processing to proceed without immediate data dependency bottlenecks, as elements can work independently in the first phase before needing to coordinate in the second phase.
Solution Approach 2:
The patent performs preliminary computation in the first iteration where processing elements compute partial results independently before the data dependency is fully realized. By completing as much computation as possible in parallel beforehand, the system reduces the impact of subsequent data routing requirements, as the remaining work in the second iteration can be completed more efficiently with the now-available intermediate results.
2Measurement precision
If reentrant compute operations are performed to resolve data dependencies, then computation accuracy is improved, but additional iterations increase processing time
Solution Approach 1:
The patent implements periodic computation with distinct iterations: a first iteration for computing partial results, followed by a second iteration for computing final results using intermediate values. This periodic structure allows the system to balance accuracy and time by performing computations in scheduled phases rather than requiring all data to be available before starting computation, thus reducing overall processing time while maintaining accuracy.
Solution Approach 2:
By performing preliminary computations in the first iteration to generate intermediate results, the system prepares data in advance for the second iteration. This preliminary action reduces the computational burden in subsequent iterations, allowing the reentrant operations to complete more quickly while still achieving the necessary computation accuracy for operations like Softmax and Layer Normalization.
Data Source
AI summary
Examples herein describe techniques for performing parallel processing using a plurality of processing elements (PEs) and a controller for data that has data dependencies. For example, a calculation may require an entire row or column to be summed, or to determine its mean. The PEs can be assigned different chunks of a data set (e.g., a tensor set, a column, or a row) for processing. The PEs can use one or more tokens to inform the controller when they are done with partial processing of their data chunks. The controller can then gather the partial results and determine an intermediate value for the data set. The controller can then distribute this intermediate value to the PEs which then re-process their respective data chunks using the intermediate value to generate final results.