Sparse Activation Convolution With Zero-Value Bypass
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks, particularly large language models, consume significant computing resources and energy due to extensive multiplication and accumulation operations, with activation maps often containing more zero values than non-zero values, leading to inefficient use of memory and processing resources.
Innovation Solution
Exploit activation sparsity by retrieving metadata to identify non-zero values in activation maps, bypassing retrieval of zero-valued portions, and performing convolution operations only on the non-zero portions, thereby conserving memory and reducing read operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If convolution operations are performed on the entire activation map including zero values, then complete computation is ensured, but memory usage and processing time increase significantly
Solution Approach 1:
The patent extracts and processes only the non-zero portions of the activation map. The system identifies regions containing non-zero values and performs convolution operations exclusively on those regions, eliminating the need to process zero-value regions. This extraction approach maintains computation completeness for relevant data while avoiding wasted operations on zero values.
Solution Approach 2:
The patent applies partial action by performing convolution operations on only a subset of the activation map - specifically, only the portions containing non-zero values. Instead of processing the entire activation map uniformly, the system selectively applies computation only where necessary, reducing overall processing time while maintaining accuracy.
2Reliability
If the entire activation map is stored in memory, then all data is available for processing, but memory consumption increases due to zero values
Solution Approach 1:
The patent extracts only the non-zero portions of the activation map for storage and processing. By identifying and storing only the relevant data segments containing non-zero values, the system significantly reduces memory consumption while ensuring that all necessary data remains available for convolution operations.
Solution Approach 2:
The patent applies local quality by treating different regions of the activation map differently based on their content. Regions containing non-zero values are stored and processed with full detail, while zero-value regions are either omitted or handled differently. This localized approach optimizes memory usage by allocating storage resources only where data is actually present.
3Reliability
If kernel coefficients are retrieved for all activation map portions, then complete convolution computation is achieved, but read operations and processing overhead increase
Solution Approach 1:
The patent extracts and retrieves kernel coefficients only for the portions of the activation map that contain non-zero values. By matching the kernel coefficient retrieval to the actual data present in the activation map, the system eliminates unnecessary read operations and reduces processing overhead while maintaining computation accuracy for the relevant regions.
Solution Approach 2:
The patent applies partial action by retrieving kernel coefficients for only a subset of the activation map regions - specifically, only those regions containing non-zero values. This selective retrieval approach reduces the number of read operations and improves processing efficiency without compromising the accuracy of the convolution computation where it is actually needed.
Data Source
AI summary
Disclosed herein are systems and methods for exploiting activation sparsity in a neural network. For example, metadata associated with at least one of a first portion or a second portion of an activation map generated by a layer of a neural network engine is retrieved. The first portion includes at least one non-zero value and the second portion includes all zero values. The first portion and a first set of kernel coefficients corresponding to the at least one non-zero value of the first portion are retrieved based on the metadata. Retrieval of the second portion and a second set of kernel coefficients corresponding to the all zero values of the second portion is bypassed based on the metadata. A convolution operation is performed by the neural network engine based on the first portion and the first set of kernel coefficients.