Activation Output Shifting and Scaling for Quantized Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Quantization of neural networks for hardware implementation results in narrower output ranges for activation functions like ReLU, leading to reduced prediction accuracy in tasks such as object detection and classification, as the sigmoid function outputs values in a narrower range than 0 to 1, affecting the minimum value output.
Innovation Solution
An information processing apparatus that shifts and scales the output of activation functions to ensure the minimum value is equal to or larger than a predetermined value, expanding the output range to align with the sigmoid function's 0 to 1 range, thereby improving prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If quantization is applied to neural networks for hardware implementation, then processing speed is improved, but output range of activation functions becomes narrower
Solution Approach 1:
The patent changes the parameters of the activation function output by introducing shifting and scaling operations. The shifted output is calculated as (output - min_value) and the scaled output is calculated as shifted_output / (max_value - min_value), transforming the quantized output range to match the sigmoid function's expected input range of 0 to 1.
2Device complexity
If activation function output range is limited, then hardware implementation is simplified, but prediction accuracy is reduced
Solution Approach 1:
The patent introduces intermediate processing steps (shifting and scaling operations) between the activation function output and the sigmoid function input. These intermediate operations act as mediators that transform the limited hardware-friendly output range into the appropriate input range for the sigmoid function, thereby maintaining prediction accuracy while preserving hardware implementation benefits.
3Ease of manufacture
If minimum value of activation function output is increased, then hardware implementation is easier, but sigmoid function output range becomes narrower
Solution Approach 1:
The patent applies parameter transformation through shifting (subtracting min_value) and scaling (dividing by max_value - min_value) to ensure the activation function output matches the sigmoid function's expected input range of 0 to 1, thereby maintaining full output versatility while preserving hardware implementation simplicity.
Data Source
AI summary
A shifting unit shifts an output of an activation function corresponding to an input, based on an output range of the activation function. A scaling unit scales the output of the activation function, the output of the activation function having been shifted by the shifting unit. An output unit outputs an output value corresponding to the output of the activation function, the output of the activation function having been scaled by the scaling unit. The activation function is a function in which the minimum value of the output of the activation function corresponding to the input is equal to or larger than a predetermined value.