Neural Network ReLU Sign Prediction for Early Operation Skipping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks using the rectified linear unit (ReLU) activation function face inefficiencies due to reduced computational savings when negative input values are correctly identified, limiting the reduction in operation amount without affecting accuracy.
Innovation Solution
A method and apparatus that set limit data based on input and weight operations to predict the sign of final resultant values, allowing early output of 0 when negative, using inverted two's complement representation for weights and bit serial operations to minimize computational load while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the ReLU activation function outputs 0 for negative input values, then the computational amount is reduced, but the degree of operation reduction is limited when negative values are correctly identified
Solution Approach 1:
The patent performs preliminary determination of the sign of the final resultant value before completing the full convolution operation. By using limit data comparison with intermediate results, the system can predict whether the final result will be negative and output 0 immediately, avoiding completion of unnecessary computational operations while maintaining accuracy.
2Productivity
If early output of 0 is performed when negative final resultant values are predicted, then computational operations are reduced, but accuracy may be affected
Solution Approach 1:
The patent uses feedback mechanisms by comparing intermediate convolution results with pre-calculated limit data to determine whether the final result will be negative. This feedback loop allows the system to make accurate predictions about the final sign based on intermediate states, enabling early termination only when confidence is high, thus maintaining accuracy while reducing operations.
3Productivity
If limit data is set based on first operation results, then subsequent operations can be skipped for negative cases, but additional computational overhead is introduced for setting limit data
Solution Approach 1:
The limit data is calculated in advance during a first operation phase using training data or representative samples. This preliminary computation stores threshold values that can be quickly compared against intermediate results in subsequent operations. The one-time overhead of setting limit data is amortized across many inference operations, where the comparison and early termination provide significant speedups.
Data Source
AI summary
A processor-implemented method of processing neural network data includes: setting first limit data by performing a first operation based on first input data and weight data generated from weights included in a filter; comparing the first limit data with an intermediate result of a second operation performed based on second input data and the weight data; and determining whether to perform a subsequent second operation based on a result of the comparing.