Dynamic Fixed-Point Adjustment in Neural Network Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing method of acquiring statistical information about the distribution of effective digits in each layer of a deep neural network (DNN) to adjust the fixed-point position for maintaining computation accuracy increases computation amount, leading to longer learning processing time due to repeated computations across layers.
Innovation Solution
An information processor that reduces computation amount while maintaining accuracy by acquiring and adjusting statistical information about the distribution of intermediate data in each layer, allowing for dynamic adjustment of the fixed-point position based on statistical information, thereby optimizing the decimal point position for improved computation efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If statistical information is acquired in each layer of the DNN to adjust the fixed-point position, then computation accuracy is maintained, but computation amount increases and learning processing time increases
Solution Approach 1:
The patent applies preliminary action by acquiring statistical information about the distribution of intermediate data values in advance during the learning processing, before the actual DNN computation. This statistical information is used to pre-determine the fixed-point position for each layer, allowing the system to maintain computation accuracy while avoiding the need to acquire statistical information repeatedly during learning processing, thus reducing computation amount and processing time
2Use of energy by moving object
If fixed-point computation is used instead of floating-point computation, then power consumption is reduced and circuit dimension is reduced, but dynamic range is narrowed and computation accuracy decreases
Solution Approach 1:
The patent applies dynamics by making the fixed-point position variable rather than fixed. The system dynamically adjusts the fixed-point position for each layer based on the statistical information about the distribution of intermediate data values. This dynamic adjustment allows the fixed-point computation to adapt to the actual data characteristics, maintaining computation accuracy while preserving the power consumption and circuit size advantages of fixed-point arithmetic
Solution Approach 2:
The patent applies parameter changes by modifying the fixed-point position parameter based on statistical information. Instead of using a static fixed-point configuration, the system changes the fixed-point position parameter according to the actual distribution characteristics of intermediate data in each layer, thereby optimizing the balance between dynamic range and computation accuracy
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An information processor comprising: in computation in a plurality of layers of a neural network for performing deep learning, an acquisition unit configured to acquire first statistical information about distribution of most significant bit position that is not a sign or least significant bit position that is not zero for each of a plurality of first fixed-point number data, the data being a computation result of the computation in the first layer; an execution unit configured to execute computation on a plurality of output data of the first layer according to a predetermined rule, in the computation in the second layer; and a control unit configured to acquire second statistical information based on the predetermined rule and the first statistical information, and determine a bit range for limiting a bit width when a plurality of second fixed-point number data, the data being a computation result of the computation in the second layer, are stored in a register, based on the second statistical information.