Convolution Activation Sparsity With Top-K Workload Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network architectures fail to effectively balance workload among processing entities during convolution operations, leading to inefficiencies and poor scalability due to unstructured activation sparsity, which results in imbalanced workload distribution.
Innovation Solution
A method and system for workload-balanced activation sparsity in neural networks, involving segmentation of input and weight tensors into sub-tensors and groups, followed by parallel processing and top-K sorting to distribute non-zero output values evenly across processors, thereby constructing a sparse and balanced output tensor.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If unstructured activation sparsity is used in convolution operations, then computational efficiency is improved by reducing the number of operations, but workload balance among processors deteriorates leading to poor scalability
Solution Approach 1:
The patent segments the activation tensor into multiple blocks and distributes them across processors. Each processor handles a specific block, ensuring that the workload is divided into manageable segments. This segmentation allows for balanced distribution of computational tasks while maintaining the benefits of sparsity-induced efficiency.
Solution Approach 2:
The patent applies different sparsity patterns to different blocks of the activation tensor based on local characteristics. By analyzing the sparsity distribution in each block and applying tailored processing strategies, the system maintains computational efficiency while ensuring balanced workload across processors with different capabilities or loads.
2Use of energy by stationary object
If activation sparsity is introduced to reduce computational cost, then energy consumption is reduced, but workload distribution among processors becomes imbalanced
Solution Approach 1:
The patent performs preliminary analysis of the activation tensor to identify sparsity patterns and block structures before distributing work to processors. By pre-processing the tensor to determine optimal block assignments and sparsity configurations, the system ensures both energy efficiency through sparsity and balanced workload distribution across processors.
Solution Approach 2:
The patent incorporates feedback mechanisms where processors report their workload status and performance metrics. Based on this feedback, the system dynamically adjusts the distribution of activation blocks and sparsity patterns to maintain balanced workload while continuing to leverage sparsity for energy efficiency.
3Speed
If parallel processing framework is used to perform convolution operations, then processing speed is improved, but system scalability deteriorates due to workload imbalance
Solution Approach 1:
The patent divides the convolution computation into independent block-level tasks that can be executed in parallel. Each processor receives a segmented portion of the activation tensor and processes it independently, enabling true parallel execution while maintaining scalability. The segmentation strategy ensures that parallel processing speedup is achieved without creating bottlenecks that would limit system scalability.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for convolution with workload-balanced activation sparsity are described. An exemplary method comprises: assigning an input tensor and a weight tensor at a convolution layer into a plurality of processors to perform Multiply-Accumulate (MAC) operations in parallel based on the input tensor and the weight tensor; obtaining a plurality of output values based on results of the MAC operations; constructing one or more banks of output values based on the plurality of output values; for each of the banks, performing a top-K sorting on the one or more output values in the bank to obtain K output values; pruning each of the banks by setting the one or more output values other than the obtained K output values in the each bank as zeros; and constructing an output tensor of the convolution layer based on the pruned banks.