Neural Processor Activation Compression for Zero-Element Elimination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural processing devices face challenges in minimizing resource waste and latency, particularly due to unnecessary multiplication calculations performed even when input data elements are zero.
Innovation Solution
The proposed neural processing device incorporates an activation compressor that generates compressed input activations by pushing effective elements from one input activation to ineffective elements in another, thereby reducing unnecessary calculations and optimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the tensor unit performs multiplication operations on all input data elements, then the calculations are completed, but resource waste occurs when input elements are zero
Solution Approach 1:
The activation compressor performs preliminary action by compressing the input activation data before it reaches the tensor unit. It identifies and removes zero elements in advance, so that when the tensor unit processes the data, it only needs to perform multiplication operations on non-zero elements, thereby avoiding resource waste while maintaining calculation completion.
Solution Approach 2:
The activation compressor extracts and removes zero elements from the input activation data. By taking out these ineffective elements before the multiplication operation, the system reduces the number of unnecessary calculations the tensor unit must perform, thus reducing resource consumption without affecting the final calculation result.
2Loss of energy
If the activation compressor compresses input activations by pushing effective elements, then resource waste is minimized, but the device complexity increases
Solution Approach 1:
The activation compressor merges the compression function with the existing neural processing device architecture. Instead of being a completely separate complex system, it integrates the element pushing and zero removal operations into the data flow path between memory and the tensor unit, thereby reducing overall device complexity while still achieving resource waste minimization.
Solution Approach 2:
The activation compressor acts as an intermediary component between the memory storage and the tensor unit. It sits in the data path, performing compression operations on input activations before they are consumed by the tensor unit. This intermediary role allows it to reduce resource waste without requiring fundamental changes to the core tensor processing functionality.
3Loss of time
If the tensor unit performs multiplication operations on compressed input activations, then latency is minimized, but the complexity of data processing increases
Solution Approach 1:
The activation compressor performs preliminary compression action on input activations before they are processed by the tensor unit. By pre-compressing the data and removing zero elements in advance, the tensor unit receives already-optimized data, which reduces the processing time and latency without significantly increasing the complexity of the tensor unit's own operations.
Solution Approach 2:
The data processing task is segmented into two distinct stages: compression stage (performed by the activation compressor) and calculation stage (performed by the tensor unit). This segmentation allows each component to specialize in its function, with the compressor handling data optimization and the tensor unit handling multiplication operations, thereby reducing overall latency while managing processing complexity through functional division.
Data Source
AI summary
A neural processing device is provided. The neural processing device comprises: an activation buffer in which first and second input activations are stored, an activation compressor configured to generate a first compressed input activation by using the first and second input activations, and a tensor unit configured to perform two-dimensional calculations using the first compressed input activation, wherein the first compressed input activation comprises first input row data comprising at least a portion of the first input activation and at least a portion of the second input activation, and first metadata corresponding to the first input row data.