Deep Neural Network Quantization Gradient Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing convolutional neural networks (CNNs) face challenges in maintaining learning accuracy when activations are quantized to reduce memory usage, as simple bit reduction degrades performance.
Innovation Solution
An information processing method that determines optimal quantization conditions for activations by calculating a gradient difference during the learning process, allowing for efficient memory usage while securing learning accuracy through the determination of quantization width and range.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If activations are quantized to reduce the number of bits, then memory capacity is saved, but learning accuracy is degraded
Solution Approach 1:
The patent changes the quantization parameters (bit width, quantization range) dynamically based on the gradient characteristics of each layer. By analyzing the gradient distribution and adjusting quantization parameters accordingly, the system maintains learning accuracy while achieving memory compression. This is accomplished through automatic quantization parameter adjustment mechanisms that adapt to different layer requirements during the learning process.
2Quantity of substance
If simple bit reduction is applied to activations, then memory usage is reduced, but the correlation between quantization and learning accuracy is lost
Solution Approach 1:
The patent implements a feedback mechanism where the learning accuracy is continuously monitored and used to adjust quantization parameters. The system evaluates the impact of quantization on gradient flow and learning performance, then dynamically adjusts the quantization bit width and range to maintain the correlation between quantization quality and learning accuracy. This closed-loop control ensures that memory reduction does not compromise the fundamental relationship between quantization precision and model performance.
Data Source
AI summary
According to one embodiment, an information processing method includes performing, in an intermediate layer of a deep neural network, a forward propagation using a first parameter and based on a first input value represented by a first bit number; performing quantization to produce a second input value represented by a second bit number smaller than the first bit number, and storing the produced second input value in the memory; calculating a second parameter based on a result of an operation using the second input value stored in the memory and a value obtained by the forward propagation, the second parameter being an update of the first parameter and for use in the learning process; and determining a condition for the quantization based on a gradient difference obtained in said calculating the second parameter.