Method for training quantized convolutional neural network by applying trend-based progressive scaling suitable for hardware accelerators

Abstract

Provided is a method for training a quantized CNN by applying trend-based progressive scaling optimized for hardware accelerators. The method for training a quantized CNN according to an embodiment of the present invention determines a next suitable scale by identifying a previous trend, and trains the quantized CNN by applying a scale factor considering clipping loss sensitivity and progressively adjusting the scale factor on the basis of previous overflow / underflow. Accordingly, training based on hardware accelerators can be performed quickly and with minimal performance degradation, and training performance can be improved by facilitating higher precision utilization than before.

Description

Quantized Convolutional Neural Network Training Method with Trend-Based Gradual Scaling Suitable for Hardware Accelerators

[0001] The present invention relates to deep learning model training based on a hardware accelerator, and more specifically, to a CNN training method that applies quantization scale adjustment suitable for a hardware accelerator.

[0002] The majority of conventional quantization-based object classification and detection learning models in computer vision are software-based. Consequently, dynamic scaling through real-time parameter distribution analysis—a common quantization technique—is unsuitable for hardware accelerator-based learning environments because it requires a separate process to verify the parameter distribution for each layer. Conversely, manual scaling fails to properly reflect the trends of overall dynamic learning, raising concerns about performance degradation.

[0003] Furthermore, as designs are required to account for the small size of training models and the constraints of limited memory space demanded by mobile or edge devices, technologies regarding computational load and complexity in the process, as well as performance, must be considered; however, the design technologies that satisfy these goals have the aforementioned problems.

[0004] The present invention has been devised to solve the above-mentioned problems, and the objective of the present invention is to provide a quantized CNN training method suitable for hardware accelerators, which involves identifying previous trends to determine the next suitable scale, applying a scale factor that considers clipping loss sensitivity, and gradually adjusting the scale factor based on previous over / under flow.

[0005] A quantized CNN learning method according to an embodiment of the present invention for achieving the above objective comprises: a step of generating an output feature map by convolutionally multiplying a quantized weight and a quantized input feature map for each layer of a CNN to be trained; a step of inversely quantizing the generated output feature map; a step of analyzing the inversely quantized output feature map to determine a quantization scale value of the output feature map; and a step of quantizing the output feature map with the determined scale value. The determination step comprises: a first setting step of setting an initial scale value for output values; a step of storing only the maximum value among the absolute values ​​of output values ​​constituting the output feature map for each batch; and a second setting step of setting a scale value for output values ​​in the next epoch based on the stored maximum values ​​whenever an epoch ends.

[0006] The determination step further includes a step of accumulating and summing the stored maximum values ​​and storing the accumulated value whenever each batch is finished; and the second setting step may calculate the average value of the stored accumulated value whenever an epoch is finished and set it as a scale value for the output values ​​in the next epoch.

[0007] The storage step may involve storing only the maximum value while updating the absolute value of the output to the maximum value during each batch execution.

[0008] The determination step may include a step of finally determining the scale value by applying a scale factor to the scale value set in the second setting step.

[0009] The scale factor can be determined based on the number of parameters of the output feature map.

[0010] The number of parameters of the output feature map can be the logarithm of the product of the batch size, height, width, and number of channels of the output feature map.

[0011] The scale factor may be the value obtained by dividing the number of parameters of the output feature map for each layer by the maximum value among the number of parameters of the output feature map for each layer.

[0012] The scale factor may include: a step of accumulating the number of overflows and underflows based on the quantization representation range during the quantization process for each batch; a step of storing the accumulated value when each batch ends; a step of calculating the average value of the accumulated value when an epoch ends; a step of setting an adjustment value for the scale factor in the next epoch based on the calculated average value; and a step of applying the set adjustment value to the scale factor.

[0013] The adjustment value of the scale factor can be set to one of the following: a value for increasing the scale factor, a value for decreasing the scale factor, or a value for maintaining the scale factor.

[0014] According to another aspect of the present invention, a CNN computation accelerator is provided, comprising: a computer that generates an output feature map by convolutionally multiplying a quantized weight and a quantized input feature map for each layer of a CNN to be trained; an inverse quantizer that inversely quantizes the generated output feature map; a scaler that analyzes the inversely quantized output feature map to determine a quantization scale value of the output feature map; and a quantizer that quantizes the output feature map with the determined scale value; wherein the scaler stores only the maximum value among the absolute values ​​of the output values ​​constituting the output feature map for each batch, and sets the scale value for the output values ​​in the next epoch based on the stored maximum values ​​whenever an epoch ends.

[0015] According to another aspect of the present invention, a method for determining a quantization scale value is provided, characterized by comprising: a step of storing only the maximum value among the absolute values ​​of output values ​​constituting an output feature map for each layer of a CNN to be trained and for each batch; and a step of setting the scale value for the output values ​​in the next epoch based on the stored maximum values ​​whenever an epoch ends.

[0016] According to another aspect of the present invention, a CNN computation accelerator is provided, characterized by comprising: a scaler that stores only the maximum value among the absolute values ​​of output values ​​constituting an output feature map for each batch for each layer of a CNN to be trained, and sets the scale value for the output values ​​in the next epoch based on the stored maximum values ​​whenever an epoch ends; and a quantizer that quantizes the output feature map with the determined scale value.

[0017] As described above, according to the embodiments of the present invention, by identifying the previous trend to determine the next suitable scale and applying a scale factor that considers clipping loss sensitivity, and by gradually adjusting the scale factor based on the previous over / under flow to train a quantized CNN, it is possible to perform fast, hardware accelerator-based training with minimal performance degradation and improve training performance by inducing the use of higher precision than before.

[0018] Fig. 1. Example of a problem occurring in the hardware structure when applying a quantization learning model in software.

[0019] Fig. 2. Average-based scale based on the maximum absolute value of each batch unit

[0020] Fig. 3. Relative clipping loss sensitivity for each layer

[0021] Fig. 4. Clipping loss sensitivity-based scale factor

[0022] Fig. 5-6. Incremental scale factor reflecting clipping loss sensitivity and learning tendency

[0023] Fig. 7. Visualization of the effect of incremental scale factor in quantization learning

[0024] FIG. 8. CNN quantization learning method according to an embodiment of the present invention

[0025] FIG. 9. CNN computation accelerator according to another embodiment of the present invention

[0026] The present invention will be described in more detail below with reference to the drawings.

[0027] As the demand for various object recognition via cameras gradually expands to the individual level, technological research for quantization models capable of real-time learning on mobile or edge devices is being conducted in various fields. Lightweighting is also a major challenge in implementing learning models suitable for NPU hardware accelerators. However, most existing quantization models implemented on hardware are executed after training and quantization have already been completed, and there are several problems in directly applying existing software quantization learning methods to implement quantization learning models suitable for hardware structures.

[0028] Figure 1 illustrates the most common process for quantization learning in software and the problems that arise when this method is applied directly to a hardware accelerator. When quantizing the convolution operation part in a CNN model, the quantization scale of the input feature map before the convolution and the quantization scale of the output feature map after the convolution have different values ​​and distributions due to intermediate operations. In the case of the output feature map scale, dynamic value analysis and comparison processes are required to quantize parameters based on an accurate value distribution.

[0029] However, when performing dynamic analysis of output feature map values ​​in real-time on hardware, all output values ​​after the convolution operation must be checked; consequently, an additional task occurs where feature map values ​​are stored in memory once more during the comparison process. This process is adversarial to the pipeline structure designed to accelerate learning speed on hardware, and thus results in the unconditional consumption of cycles for that process. Furthermore, since the size of the feature map stored during this process is a 32-bit floating-point value, it is not suitable for mobile and edge devices that assume limited memory sizes.

[0030] Another method involves static quantization. While this method has a simpler structure due to the omission of intermediate steps compared to dynamic methods, it is primarily utilized in post-processing due to the disadvantage of not being able to verify parameter distribution in real time.

[0031] Accordingly, an embodiment of the present invention presents a quantized CNN training method that applies trend-based progressive scaling suitable for a hardware accelerator. The technique involves identifying the previous trend to determine the next suitable scale, applying a scale factor that considers clipping loss sensitivity, and training the quantized CNN by progressively adjusting the scale factor based on the previous over / under flow.

[0032] Specifically, the trend of each layer in the learning process is identified to predict and adjust the scale in the next epoch, the clipping loss sensitivity for each layer is defined, and an adjustment scale factor is applied considering the sensitivity, and the scale factor is adjusted through a previous very small number of values ​​that reflect the characteristics of the hardware accelerator.

[0033] First, the average scale calculation method, which adjusts by predicting future distributions based on previous trends without applying dynamic quantization methods during the learning process, is explained below.

[0034] To construct a training model implementable on a hardware accelerator, initial scale values ​​for the outputs of each layer are stored in advance, and training begins. During the training process, the absolute values ​​of each output are checked, and only the single maximum value among them is continuously updated and stored. At the end of each batch, the corresponding values ​​are stored in their respective buffers as cumulative sums, and the process of initializing the buffer storing the single maximum value is repeated. After the completion of one epoch of training, the average of the accumulated values ​​is calculated, and the average scale based on this value is used as the baseline for the next epoch of training.

[0035] The above method according to an embodiment of the present invention is very simple and naturally induces first-order clipping for an appropriate range during the process of calculating the average of outliers. FIG. 2 is a schematic representation of the process of calculating an average-based scale according to the above. For comparison, the process of calculating a scale using only the maximum value in epoch units is illustrated on the left side of FIG. 2.

[0036] In an embodiment of the present invention, relative clipping loss sensitivity for each layer is additionally applied to the scale. Clipping loss sensitivity is determined based on the number of parameters of the output feature map for each layer. The parameters in each layer represent the representation results in that layer, and as shown in FIG. 3, the performance of the overall model training is affected depending on how much the precision of this representation is optimized.

[0037] Accordingly, the relative clipping loss sensitivity to be reflected in the embodiment of the present invention is based on the number of parameters of the simplest and most reliable feature map, and is based on the fact that a large-sized feature map with many surrounding nodes that can be relatively replaced has a robust structure for the same ratio of clipping situations compared to a feature map that does not.

[0038] In addition, experimental results confirmed that output feature maps close to the final layer, which output relatively few parameters, have a negative impact on learning when the scale range of the actual distribution is somewhat narrow.

[0039] Therefore, for layers with relatively few output parameters, an additional scale factor is applied by multiplying the scale value by a logarithmic ratio based on the number of relative parameters. The scale factor is shown in Fig. 4. Fig. 4 is a diagram showing a scale factor based on clipping loss sensitivity.

[0040] Furthermore, in an embodiment of the present invention, a learning incremental scale factor based on clipping loss sensitivity values ​​is proposed to additionally reflect fine-tuning values ​​through prediction of learning trends. The incremental scale factor is finally multiplied by the average-value-based scale described in FIG. 2 and used as the scale value for the next epoch.

[0041] The incremental scale factor accumulates the number of overflows and underflows based on the quantization representation range during the quantization process in each layer during training. This accumulation is performed on a batch basis, and, similar to the average scale, the average of the accumulated values ​​is calculated after each epoch of training is completed. The calculated average value determines whether the scale value used in the previous epoch was appropriate based on a parameter count flag derived from the clipping loss sensitivity ratio for each layer. The flag criteria consist of a total of three cases; if the number exceeds a certain threshold, the scale factor to be applied in the next epoch is finely adjusted upwards; if it falls below that threshold, it is finely adjusted downwards; and if it is maintained at an appropriate ratio, training proceeds without further fine-tuning. The detailed computation process for the incremental scale factor is presented in Fig. 5, and the fine-tuning of the scale factor through this is exemplified in Fig. 6.

[0042] The average scale based on the progressive scale factor presented in the embodiment of the present invention achieves high model training performance without real-time dynamic scale updates to maintain the advantages of hardware accelerators. Furthermore, the embodiment of the present invention maintains high training performance by maintaining optimal distribution representation for each layer by considering clipping loss sensitivity. Moreover, as a solution to the static representation problem that may occur when applying these configurations, the embodiment of the present invention presents a progressive scale factor that identifies the trend of previous training and induces optimal representation at every epoch. The progressive scale factor is a technique based on clipping loss sensitivity rather than a simple operation of a fixed ratio, and as shown in FIG. 7, it enhances stability in the overall training of the quantization model.

[0043] FIG. 8 is a diagram illustrating the flow of a CNN quantization learning method according to an embodiment of the present invention. The steps of the method illustrated in FIG. 8 are performed for each layer of the CNN to be learned.

[0044] First, initial scale values ​​are set for the output values ​​constituting the output feature map, and then the output feature map is generated by convolutionally multiplying the quantized weights and the quantized input feature map (S110). Next, the output feature map generated in step S110 is dequantized (S120), the quantization scale value of the dequantized output feature map is determined (S130), and the output feature map is quantized with the determined scale value (S140). Steps S110 through S140 are repeated until training is completed.

[0045] The step for determining the quantization scale value (S130) is described in detail below.

[0046] To determine the quantization scale value, first, for each batch, only the maximum value among the absolute values ​​of the output values ​​constituting the output feature map must be stored. To do this, while performing each batch, the absolute value of the output value is updated to the maximum value, and only the maximum value among the absolute values ​​of the output values ​​for each batch is stored (S131).

[0047] And when each batch ends, the accumulated maximum value is stored by summing the stored maximum values ​​(S132), and when an epoch ends, the average value of the accumulated maximum values ​​is calculated (S133) and set as the quantization scale value for the next epoch (S134).

[0048] Meanwhile, for each batch, the number of overflows and underflows based on the quantization representation range during the quantization process is stored, and the accumulated number is stored by summing the stored number whenever each batch ends (S135). Then, whenever an epoch ends, the average of the accumulated number is calculated (S136), and based on the calculated average value, the adjustment value of the scale factor for the next epoch is set (S137). The adjustment value of the scale factor is set as one of a value for adjusting the scale factor upward, a value for adjusting the scale factor downward, or a value for maintaining the scale factor.

[0049] Then, the scale factor according to the formula in Fig. 4, which is the value obtained by dividing the number of parameters of the output feature map (log value of the product of the batch size, height, width, and number of channels of the output feature map) by the maximum value among the number of parameters of the output feature map for each layer, is multiplied by the adjustment value set in step S137 to calculate the adjusted scale factor (S138).

[0050] The adjusted scale factor calculated in the next step S138 is multiplied by the quantization scale value set in step S134 to finally determine the quantization scale value (S139).

[0051] FIG. 9 is a diagram illustrating the configuration of a CNN computation accelerator according to another embodiment of the present invention. As illustrated, the CNN computation accelerator according to an embodiment of the present invention is configured to include a computer (210), an inverse quantizer (220), a scaler (230), a quantizer (240), and a memory (250).

[0052] The operator (210) generates an output feature map by convolutionally multiplying the quantized weights and the quantized input feature map. The de-quantizer (220) de-quantizes the output feature map generated by the operator (210). The scaler (230) determines / adjusts the quantization scale value of the de-quantized output feature map according to steps S131 to S139 of FIG. 8 described above. The quantizer (240) quantizes the output feature map with the scale value determined by the scaler (230).

[0053] The memory (250) is a storage space for storing data required during the execution of the procedure illustrated in FIG. 8.

[0054] Up to now, a quantized CNN training method applying trend-based progressive scaling suitable for hardware accelerators has been described in detail with preferred embodiments.

[0055] In the above embodiment, the next suitable scale is determined by identifying the previous trend, and a scale factor considering clipping loss sensitivity is applied. By training a quantized CNN by gradually adjusting the scale factor based on the previous over / under flow, it is possible to perform fast, hardware accelerator-based training with minimal performance degradation, and to improve training performance by inducing the use of higher precision than before.

[0056] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

Claims

1. A step of generating an output feature map by convolutionally multiplying the quantized weights and the quantized input feature maps for each layer of the CNN to be trained; A step of inversely quantizing the generated output feature map; A step of analyzing an inversely quantized output feature map to determine the quantization scale value of the output feature map; The step of quantizing the output feature map with a determined scale value; is included, The decision stage is, A first setting step for setting initial scale values ​​for output values; For each batch, a step of storing only the maximum value among the absolute values ​​of the output values ​​constituting the output feature map; A quantization CNN learning method characterized by including: a second setting step of setting scale values ​​for output values ​​in the next epoch based on stored maximum values ​​whenever an epoch ends.

2. In Claim 1, The decision stage is, It further includes the step of accumulating the maximum value stored and storing the accumulated value whenever each batch ends; The second setting step is, A quantization CNN training method characterized by calculating the average value of the stored accumulated values ​​at the end of each epoch and setting it as a scale value for the output values ​​in the next epoch.

3. In Claim 2, The storage step is, A quantization CNN training method characterized by updating the absolute value of the output value to the maximum value and storing only the maximum value during each batch execution.

4. In Claim 1, The decision stage is, A quantization CNN learning method characterized by including a step of finally determining a scale value by applying a scale factor to a scale value set in a second setting step.

5. In Claim 4, The scale factor is, A quantization CNN training method characterized by being determined based on the number of parameters of an output feature map.

6. In Claim 5, The number of parameters of the output feature map is, A quantization CNN training method characterized by being the logarithm of the product of the batch size, height, width, and number of channels of the output feature map.

7. In Claim 6, The scale factor is, A quantization CNN learning method characterized by the value obtained by dividing the number of parameters of the output feature map for each layer by the maximum value among the number of parameters of the output feature map for each layer.

8. In Claim 4, The scale factor is, For each batch, a step of accumulating the number of overflows and underflows based on the quantization representation range during the quantization process in progress during training; A step of storing the accumulated value whenever each batch ends; A step of calculating the average value of the accumulated values ​​whenever an epoch ends; A step of setting an adjustment value for the scale factor in the next epoch based on the calculated average value; A quantization CNN training method characterized by including the step of applying a set adjustment value to a scale factor.

9. In Claim 8, The adjustment value of the scale factor is, A quantization CNN training method characterized by being set to one of a value for up-adjusting the scale factor, a value for down-adjusting the scale factor, or a value for maintaining the scale factor.

10. An operator that generates an output feature map by convolutionally multiplying quantized weights and quantized input feature maps for each layer of the CNN to be trained; Inverse quantizer that inversely quantizes the generated output feature map; A scaler that analyzes an inversely quantized output feature map and determines the quantization scale value of the output feature map; quantizer for quantizing an output feature map to a determined scale value; The scaler is, A CNN computation accelerator characterized by storing only the maximum value among the absolute values ​​of the output values ​​constituting the output feature map for each batch, and setting the scale value for the output values ​​in the next epoch based on the stored maximum values ​​whenever an epoch ends.

11. A step of storing only the maximum value among the absolute values ​​of the output values ​​constituting the output feature map for each layer of the training CNN and for each batch; A method for determining a quantization scale value, characterized by including the step of setting a scale value for the output values ​​in the next epoch based on the stored maximum values ​​whenever an epoch ends.

12. A scaler that stores only the maximum value among the absolute values ​​of the output values ​​constituting the output feature map for each layer of the CNN to be trained and for each batch, and sets the scale value for the output values ​​in the next epoch based on the stored maximum values ​​whenever an epoch ends; A CNN computation accelerator characterized by including a quantizer that quantizes an output feature map to a determined scale value.

Images