Neural Network Activation Range Shifting for Accurate Quantization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Quantization of neural networks for hardware implementation results in reduced prediction accuracy due to activation functions outputting values in a narrower range than required, affecting tasks like object detection and classification.

Innovation Solution

An information processing apparatus that shifts and scales the output of activation functions to ensure the minimum value is equal to or larger than a predetermined value, expanding the output range to match the required 0 to 1 range for sigmoid functions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If quantization is applied to neural networks for hardware implementation, then processing speed is improved, but prediction accuracy deteriorates due to activation function output range limitations

Engineering Contradiction:

Improveprocessing speedVSAvoidprediction accuracy

Core Design Contradiction:

SpeedVSMeasurement precision

Solution Approach 1:

The patent changes the parameters of the activation function output by introducing shifting and scaling operations. The shifting operation adjusts the minimum output value from a fixed value (e.g., 0 for ReLU) to a configurable value, while the scaling operation adjusts the output range. This allows the activation function output to match the required input range of the sigmoid function (0 to 1), thereby maintaining prediction accuracy while keeping the hardware implementation benefits of quantization.

Inventive Principle:

Principle #35Parameter changes

2Device complexity

If an activation function with a limited minimum value (e.g., ReLU) is used in hardware, then device complexity is reduced, but the output range becomes narrower than required for sigmoid functions, reducing prediction accuracy

Engineering Contradiction:

Improvedevice complexityVSAvoidprediction accuracy

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The patent segments the activation function output processing into three distinct stages: (1) the original activation function computation, (2) a shifting operation to adjust the minimum value, and (3) a scaling operation to adjust the output range. This segmentation allows each stage to be implemented independently in hardware, maintaining simplicity while achieving the required output characteristics for accurate prediction.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces intermediate processing operations (shifting and scaling) between the activation function and the sigmoid function. These intermediate operations act as mediators that transform the activation function output into a format suitable for the sigmoid function, ensuring compatibility and maintaining accuracy without requiring complex redesign of either component.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12469264B2Information processing apparatus, image capturing apparatus, method, and non-transitory computer readable storage medium

Publication Date: 2025.11.11 CANON KK

AI summary

A shifting unit shifts an output of an activation function corresponding to an input, based on an output range of the activation function. A scaling unit scales the output of the activation function, the output of the activation function having been shifted by the shifting unit. An output unit outputs an output value corresponding to the output of the activation function, the output of the activation function having been scaled by the scaling unit. The activation function is a function in which the minimum value of the output of the activation function corresponding to the input is equal to or larger than a predetermined value.