Dynamic Processing Element Array Expansion for Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks face high latency in inference due to limited resources in underlying hardware, requiring serial performance of tensor operations which increases computation time, especially in complex models like deep neural networks.
Innovation Solution
Splitting tensor operations, such as convolution or multi-head attention, into sub-operations that can be performed in parallel by multiple computing engines, allowing each engine to generate portions of the output tensor, which are then concatenated without additional arithmetic operations to reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If tensor operations are performed serially on limited hardware resources, then device complexity is reduced, but inference latency increases
Solution Approach 1:
The patent divides tensor operations into multiple sub-operations that can be executed in parallel by different processing elements. Specifically, the convolution operation is segmented into multiple partial convolutions that operate on different portions of the input data, allowing simultaneous execution and reducing overall inference latency without requiring a complete overhaul of the hardware architecture
Solution Approach 2:
The patent introduces a temporal dimension to the processing architecture by enabling multiple processing elements to operate at different stages of the same tensor operation simultaneously. This creates a pipeline effect where different sub-operations progress through different stages in parallel, effectively adding a time-based dimension to the computational flow and reducing latency without increasing spatial hardware complexity
2Productivity
If more processing elements are added to increase parallelism, then productivity improves, but device complexity increases
Solution Approach 1:
The patent designs processing elements that can dynamically assume different roles based on the current operation phase. The same physical processing element can function as an input buffer in one cycle, perform computations in the next cycle, and serve as an output buffer subsequently. This multi-functionality allows the system to achieve high productivity with fewer physical elements, as each element contributes to multiple stages of the parallel processing pipeline
Data Source
AI summary
A computer-implemented method includes receiving a neural network model that includes a tensor operation, dividing the tensor operation into a set of sub-operations, and generating instructions for performing a plurality of sub-operations of the set of sub-operations on respective computing engines of a plurality of computing engines on a same integrated circuit device or on different integrated circuit devices. Each sub-operation of the set of sub-operations generates a portion of a final output of the tensor operation. An inference is made based on a result of a sub-operation of the plurality of sub-operations, or based on results of the plurality of sub-operations.