Information processing device, and information processing method

Abstract

This information processing device uses a convolutional neural network and comprises: a multiplication unit that calculates the product of an input feature map and a kernel; an accumulation unit that calculates a product sum from the product; a shift unit that bit-shifts the product sum; and an activation unit that converts the product sum after the bit-shift by using an activation function. The shift unit obtains a plurality of post-shift product sums by bit-shifting the product sum more than once while changing the shift amount. The activation unit obtains a plurality of first feature maps by executing a first quantization process including using an activation function expressing a value the absolute value of which is relatively small, for the post-shift product sum the quantization step of which is relatively small, and using an activation function expressing a value the absolute value of which is relatively large, for the post-shift product sum the quantization step of which is relatively large.

Description

Information processing apparatus and information processing method

[0001] The disclosed technologies relate to information processing devices and information processing methods.

[0002] Traditionally, the Convolutional Neural Network (CNN) has been known as a representative method in deep learning. CNNs are used in various still and moving image recognition technologies, such as object detection, semantic segmentation, and pose estimation, achieving high-precision inference.

[0003] CNNs are multi-layered network models that extract features from input data and perform tasks such as classification or regression by stacking input layers, convolutional layers, pooling layers, fully connected layers, and output layers. In particular, the convolutional layers generate feature maps by repeatedly performing a vast number of sum-of-products operations between the kernel (also called a filter) and the input data, resulting in a large amount of computation. In other words, building a CNN requires high computing power and the processing of large amounts of data.

[0004] Therefore, in recent years, dedicated LSIs (Large Scale Integrations) that possess dedicated circuits specifically designed for executing CNNs have attracted attention. Because dedicated LSIs offer low power consumption and high-speed processing, demand for them is rapidly increasing in areas such as edge devices and applications requiring real-time inference. Various AI (Artificial Intelligence) chips, including dedicated LSIs, are optimized for operations such as multiply-accumulate in CNNs, achieving faster computation.

[0005] Non-patent document 1 describes quantizing values ​​handled in a dedicated circuit for CNNs to low-precision values ​​such as 8-bit fixed-point numbers. Specifically, data input to the convolutional layer, such as the kernel and input feature map, is represented by 8 bits. In the multiply-accumulate operation performed during the convolution process, a 16-bit operation result is obtained by multiplying the 8-bit kernel and the 8-bit input feature map. By extracting 8 bits from the 16-bit multiply-accumulate result through bit shifting, an 8-bit output feature map is finally obtained.

[0006] When extracting 8 bits by bit shifting, it is necessary to determine the quantization step, that is, the value of the decimal point position represented by the 8-bit LSB (Least Significant Bit). The granularity of the quantization step can be arbitrarily set, for example, per layer, per channel, per pixel, or per value. For example, Non-Patent Literature 2 describes setting one quantization step per convolutional layer when implementing CNN convolution operations in hardware. In this case, the operations within the layer are completed only by operations on the bit width representing significant figures, thus simplifying the circuit implementation.

[0007] On the other hand, there is a trade-off between the simplicity of circuit implementation and the smallness of quantization error. When the quantization step is set for each layer, the quantization error becomes relatively large. In other words, if the granularity of setting the quantization step is finer, for example, for each value, the quantization error can be reduced, but the calculation becomes more complex and the circuit area increases. Specifically, if the quantization steps are different, it becomes necessary to align the digits during cumulative addition, or to select an appropriate quantization step according to the magnitude of the value (normalization).

[0008] Furthermore, if one attempts to set the granularity of the quantization steps finely, for example, for each value, the number of quantization steps becomes enormous. Therefore, it is desirable to use a representation method that can maintain the decimal point position within the value, such as floating-point numbers. Floating-point arithmetic is complex and contributes to an increase in circuit area. For example, Non-Patent Document 3 describes the process of implementing logarithmic arithmetic for floating-point numbers using a fixed-point arithmetic circuit, which involves exponential processing, shifting, and zero counting, and the increase in processing cycles is a challenge.

[0009] Zhisheng Li et.al, "Laius: An 8-Bit Fixed-Point CNN Hardware Inference Engine." 2017 IEEE International Symposium on Parallel and Distributed Processing with Applications and 2017 IEEE International Conference on Ubiquitous Computing and Communications (ISPA / IUCC). IEEE, 2017. Markus Nagel et.al, "A White Paper on Neural Network Quantization." arXiv preprint arXiv:2106.08295, 2021. Julien Le Maire et.al, "Computing floating-point logarithms with fixed-point operations." 2016 IEEE 23nd Symposium on Computer Arithmetic (ARITH). IEEE, 2016.

[0010] In recent years, there has been a demand for technologies that can reduce quantization errors while suppressing an increase in circuit area, such as setting one quantization step per convolutional layer.

[0011] The disclosed technology was developed in view of the above points, and aims to provide an information processing device and an information processing method that can achieve both reduced circuit area and reduced quantization error.

[0012] A first aspect of this disclosure is an information processing device using a convolutional neural network, comprising: a multiplication unit that calculates the product of an input feature map and a kernel; an accumulation unit that calculates the sum of products from the product; a shift unit that bit-shifts the sum of products; and an activation unit that transforms the bit-shifted sum of products using an activation function, wherein the shift unit obtains a plurality of shifted sums of products by bit-shifting the sum of products a plurality of times while changing the shift amount; and the activation unit obtains a plurality of first feature maps by performing a first quantization process which includes using an activation function that represents a value with a relatively small absolute value for the shifted sums of products with a relatively small quantization step, and using an activation function that represents a value with a relatively large absolute value for the shifted sums of products with a relatively large quantization step.

[0013] A second aspect of this disclosure is an information processing method using a convolutional neural network, the computer performing a process to obtain a plurality of first feature maps by performing a first quantization process, which includes calculating the product of an input feature map and a kernel, calculating the sum of products from the product, bit-shifting the sum of products, and transforming the bit-shifted sum of products using an activation function, wherein the bit-shifting involves bit-shifting the sum of products multiple times while changing the shift amount to obtain a plurality of shifted sums of products, and the transformation involves using an activation function that represents a value with a relatively small absolute value for the shifted sums of products with a relatively small quantization step, and using an activation function that represents a value with a relatively large absolute value for the shifted sums of products with a relatively large quantization step.

[0014] According to the disclosed technology, it is possible to provide an information processing device and an information processing method that can achieve both reduced circuit area and reduced quantization error.

[0015] This is a diagram illustrating the differences due to the quantization steps. This is a block diagram showing an example of the hardware configuration of an information processing device. This is a block diagram showing an example of the functional configuration of an information processing device. This is a diagram illustrating the processing method of the convolutional layer. This is a diagram illustrating the method for determining the shift amount. This is a diagram illustrating the method for determining the activation function. This is a diagram illustrating the bit manipulation method of the kernel. This is a flowchart showing the flow of information processing. This is a flowchart showing the flow of the first quantization process. This is a flowchart showing the flow of the second quantization process.

[0016] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals. Furthermore, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0017] [Overview of CNN-Specific Circuitry] The disclosed technology relates to a Convolutional Neural Network (CNN), a representative method in deep learning. CNNs are used in various still and moving image recognition techniques, such as object detection, semantic segmentation, and pose estimation, achieving high-precision inference.

[0018] CNNs are multi-layered network models that extract features from input data and perform tasks such as classification or regression by stacking input layers, convolutional layers, pooling layers, fully connected layers, and output layers. In particular, the convolutional layers generate feature maps by repeatedly performing a vast number of sum-of-products operations between the kernel (also called a filter) and the input data, resulting in a large amount of computation.

[0019] Conventionally, inference processing has been performed using dedicated circuits specifically designed for executing CNNs (hereinafter referred to as CNN-dedicated circuits) to achieve faster computation and reduced power consumption. In CNN-dedicated circuits, the values ​​being handled are sometimes quantized to low-precision values ​​such as 8-bit fixed-point numbers in order to improve the efficiency of memory consumption and the area efficiency of the processing unit.

[0020] Specifically, in the case of 8-bit fixed-point numbers, the data input to the convolutional layer, such as the kernel, input feature map, and bias value, is represented by 8 bits. In the multiply-accumulate operation performed during the convolution process, a 16-bit result is obtained by multiplying the 8-bit kernel and the 8-bit input feature map. By extracting 8 bits from the 16-bit multiply-accumulate result through bit shifting, an 8-bit output feature map is finally obtained.

[0021] When extracting 8 bits by bit shifting, it is necessary to determine the quantization step, that is, the value of the decimal point position represented by the 8-bit LSB (Least Significant Bit). As shown in Figure 1, the granularity of the quantization step can be arbitrarily set, for example, per layer, per channel, per pixel, or per value. As shown in Figure 1, there is a trade-off between the simplicity of circuit implementation and the smallness of quantization error. For example, if one quantization step is set for each convolutional layer, the calculation within the layer is completed only by calculating the bit width representing the significant figures, so the circuit implementation becomes simpler, but the quantization error becomes relatively large.

[0022] For example, if one quantization step is set for each value, the quantization error can be kept relatively small, but the calculation becomes relatively complex, and the circuit area also increases relatively. Specifically, because the quantization step differs for each value, digit alignment is required during cumulative addition, and it becomes necessary to select an appropriate quantization step according to the magnitude of the value (normalization). Also, because the number of quantization steps to set becomes enormous, it is desirable to use a representation method that can maintain the decimal point position within the value, such as floating-point numbers. Therefore, a complex circuit design that enables floating-point calculations is required.

[0023] [Configuration of the Information Processing Device of the Disclosure] In this disclosure, the circuit area is reduced by using a simple CNN-dedicated circuit in an information processing device using a CNN, such as when one quantization step is set for one convolutional layer. Furthermore, by enabling calculations such as when multiple quantization steps are set for one convolutional layer using this CNN-dedicated circuit, the quantization error is reduced. The information processing device 10 according to this embodiment will be described below.

[0024] Figure 2 is a block diagram showing an example of the hardware configuration of the information processing device 10. The information processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, storage 14, an input unit 15, a display unit 16, a communication interface 17, and a dedicated CNN circuit 20. Each component is connected to the others via a bus 19 so as to be able to communicate with each other.

[0025] The CPU 11 is a central processing unit that executes various programs and controls various parts. Specifically, the CPU 11 reads a program from the ROM 12 or storage 14 and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculations according to the program stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores an information processing program for executing information processing described later.

[0026] ROM 12 stores various programs and data. RAM 13 temporarily stores programs or data as a working area. Storage 14 consists of storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive) and stores various programs and data, including the operating system.

[0027] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used for various types of input. The display unit 16 is, for example, a liquid crystal display and displays various types of information. The display unit 16 may also function as the input unit 15 by employing a touch panel system.

[0028] Communication I / F 17 is an interface for communicating with other devices. For this communication, a wired communication standard such as Ethernet (registered trademark) or FDDI (Fiber Distributed Data Interface), or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) can be used. As the information processing device 10, for example, a personal computer and a server computer can be appropriately applied.

[0029] The CNN-dedicated circuit 20 performs CNN processing such as convolution and activation function processing. Specifically, the CNN-dedicated circuit 20 is optimized for multiply-accumulate (MAC) operations in convolution processing, resulting in faster computation. Furthermore, the CNN-dedicated circuit 20 is designed to perform CNN processing when one quantization step is set for each convolutional layer. In other words, the CNN-dedicated circuit 20 has a simpler design than the circuits required when setting quantization steps finely, such as for each value, thus reducing the circuit area.

[0030] The CNN dedicated circuit 20 is configured as hardware designed specifically for CNN, such as various AI (Artificial Intelligence) chips like a dedicated LSI (Large Scale Integration). The CNN dedicated circuit 20 may be configured as a dedicated electric circuit that is a processor having a circuit configuration designed specifically for executing specific processes, such as a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit). Also, the CNN dedicated circuit 20 may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, and a combination of an ASIC and an FPGA, etc.). Also, the hardware structure of these various processors is, more specifically, an electric circuit combining circuit elements such as semiconductor elements.

[0031] FIG. 3 is a block diagram showing an example of the functional configuration of the information processing apparatus 10. The information processing apparatus 10 includes, as functional configurations, a product-sum calculation unit 22, a value range control unit 26, an activation unit 28, and a control unit 30. The product-sum calculation unit 22 includes a multiplication unit 23, an accumulation unit 24, and a shift unit 25, and performs a product-sum calculation with these functional configurations. The functional configurations of the product-sum calculation unit 22, the value range control unit 26, and the activation unit 28 are realized by the CNN dedicated circuit 20. A memory 27 such as an SRAM (Static RAM) is mounted on the CNN dedicated circuit 20. The functional configuration of the control unit 30 is realized by the CPU 11 reading an information processing program stored in the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it. In FIG. 3, the feature map is referred to as "Fmap".

[0032] The control unit 30 acquires an input image (video) or an input feature map and a kernel to be input to the convolutional layer, and inputs them to the CNN dedicated circuit 20. The input feature map and the kernel may have a plurality of input channels. The input image (video), the input feature map, and the kernel are stored, for example, in the ROM 12, the storage 14, or an external storage device or the like.

[0033] The multiplication unit 23 in the CNN dedicated circuit 20 calculates the product of the input feature map and the kernel, and outputs it to the accumulation unit 24. When the input feature map and the kernel have a plurality of input channels, the multiplication unit 23 calculates the product of the input feature map and the kernel for each input channel, and outputs each product to the accumulation unit 24.

[0034] The accumulation unit 24 calculates (accumulatively adds) the sum of products in the channel direction for the product of the input feature map and the kernel. Specifically, when the accumulation unit 24 first receives a product, it temporarily stores the received product in the memory 27. When the accumulation unit 24 receives the product for the next input channel, it reads out the product stored in the memory 27, calculates the sum of products at that time by adding the new product to the read product, and temporarily stores the sum of products in the memory 27. When the accumulation unit 24 further receives the product for the next input channel, it reads out the sum of products stored in the memory 27, calculates the sum of products at that time by adding the new product to the read sum of products, and updates the value of the sum of products temporarily stored in the memory 27. The accumulation unit 24 repeats the update of the sum of products for the number of input channels, and when the calculation of the sum of products for the input channels is completed, it outputs to the shift unit 25.

[0035] After the accumulation unit 24 outputs the sum of products in the channel direction, it may initialize the sum of products temporarily stored in the memory 27. In the accumulation unit 24, whether to initialize the sum of products temporarily stored in the memory 27, that is, the timing of initialization, is configured to be arbitrarily set.

[0036] The shift unit 25 performs a bit shift on the sum of products. Specifically, the shift unit 25 performs a bit width reduction process by performing a right bit shift on the sum of products whose bit width has expanded beyond the input feature map and kernel due to multiplication. The shift unit 25 may add a bias value to the sum of products before or after the bit shift. The shift unit 25 may perform a saturation process on the sum of products after the bit shift by clipping to converge values ​​outside a predetermined range (for example, values ​​with large absolute values) to the maximum or minimum value.

[0037] The range control unit 26 specifies the amount of bit shift in the shift unit 25. Specifically, the range control unit 26 determines the shift amount for each layer. That is, there is only one shift amount per layer, and it does not differ for, for example, channels, pixels, and values. Details of the range control unit 26 will be described later.

[0038] The activation unit 28 obtains a feature map by transforming the sum of products after bit shifting using an activation function. The activation function is, for example, a normalized linear function (ReLU: Rectified Linear Unit). The activation unit 28 is configured to allow the use of different activation functions for each layer.

[0039] The activation unit 28 outputs the feature map to the control unit 30. The control unit 30 stores this feature map in, for example, ROM 12, storage 14, or an external storage device, so that it can be used as an input feature map for the next layer.

[0040] Referring to Figure 4, the processing method for the convolutional layer in the information processing device 10 according to this embodiment will be described. In this embodiment, multiple inference paths with different quantization steps are set in a single convolutional layer. This makes it possible to perform calculations similar to those performed when multiple quantization steps are set, while using a simple CNN-dedicated circuit 20, similar to the case where one quantization step is set per convolutional layer.

[0041] To explain the processing using multiple inference paths, in this embodiment, one convolutional layer is considered to have a two-layer structure: an X layer that performs the first quantization process and a Y layer that performs the second quantization process. In the following description, "X layer" and "Y layer" refer to a series of processing units such as multiplication, accumulation, shifting, and activation for a single input in the CNN-dedicated circuit 20, and do not correspond to general layers in CNNs such as convolutional layers and pooling layers.

[0042] The X layer has a k-layer structure consisting of X1, X2, ..., Xk layers. In the X layer, one input feature map is used for one convolutional layer, and k feature maps are obtained by performing calculations using the CNN-dedicated circuit 20 in each layer. Each layer has a different quantization step and activation function set, which allows the feature map to be divided into k parts according to the magnitude of the values. Hereafter, the processing of the entire X layer will be collectively referred to as the "first quantization process," and the feature map obtained by the first quantization process will be referred to as the "first feature map." In other words, k first feature maps are obtained by performing the first quantization process once on the entire X layer.

[0043] The Y layer has a k-layer structure, with the Y1 layer receiving the first feature map of the X1 layer, the Y2 layer receiving the first feature map of the X2 layer, ..., and the Yk layer receiving the first feature map of the Xk layer. In the Y layer, the cumulative function of the CNN dedicated circuit 20 is used to cumulatively add the k first feature maps output from the X layer to obtain one feature map. The feature map output from the Y layer becomes the final output feature map of this convolutional layer, and after processing in the pooling layer, etc., it becomes the input feature map for the next convolutional layer. Hereafter, the processing of the entire Y layer will be collectively referred to as the "second quantization process". That is, one output feature map is obtained by performing the second quantization process once on the entire Y layer.

[0044] This layer structure allows for the creation of k inference paths within a single convolutional layer: inference path 1 using layers X1 and Y1 as the processing path, inference path 2 using layers X2 and Y2 as the processing path, ..., inference path k using layers Xk and Yk as the processing path. The number of divisions k for layers X and Y is an integer of 2 or more and can be determined arbitrarily. The number of divisions k may be the same for each convolutional layer, or it may be different for each convolutional layer. That is, the number of each layer in the X layer and the number of each layer in the Y layer may differ for each convolutional layer. For example, the number of divisions k may be determined according to the required accuracy for each convolutional layer.

[0045] [First Quantization Process] The first quantization process in layer X is explained below. In the first quantization process, quantization is performed while changing the quantization step in each of the X1, X2, ..., Xk layers. By changing the quantization step for each layer, it is possible to prevent the k first feature maps from overlapping. In other words, it is possible to prevent different first feature maps from having the same value, thus enabling normalization.

[0046] The range control unit 26 determines the shift amount for each layer according to the range of the next convolutional layer and the quantization step of each layer, X1, X2, ..., Xk, and provides it to the shift unit 25. The quantization step of each layer is predetermined according to, for example, the required accuracy and computational efficiency for each convolutional layer.

[0047] Referring to Figure 5, an example of how to determine the different shift amounts for each layer is explained. Figure 5 shows how a 16-bit sum of products is reduced to 8 bits by a right bit shift. In Figure 5, one bit is represented by one rectangle, and the numbers from -2 to 13 in the top row indicate the bit weight. In this example, the range of the next convolutional layer is from -128 to 127. The number of divisions k for the X layer is 2, and it includes the X1 and X2 layers. The sum of products before the shift is represented as a 16-bit fixed-point number, and the sum of products after the shift is represented as an 8-bit fixed-point number. The quantization step for the X1 layer is 2 －２ Therefore, the quantization step for the X2 layer is 2 ０ That is the case.

[0048] In the X1 layer, the aim is to improve the precision of the values ​​by utilizing a smaller quantization step than the X2 layer. For example, by including the lower 8 bits of the 16-bit sum of products, high-precision values ​​can be represented in the range of -32 to 31.75. In this case, the range control unit 26 determines the shift amount of the right bit shift to be 0.

[0049] In the X2 layer, the aim is to match the range of values ​​of the next convolutional layer. For example, if the quantization step is 2 ０ In the 8 bits, in order to represent the range of the next convolutional layer, which is from -128 to 127, the range control unit 26 determines the amount of the right bit shift to be 2.

[0050] The shift unit 25 bit-shifts the sum of products for each layer according to the given shift amount for each layer. In other words, for the entire first quantization process, the shift unit 25 bit-shifts the sum of products multiple times while changing the shift amount, thereby obtaining multiple shifted sums of products.

[0051] Bit shifting allows the X1 layer to represent values ​​with small absolute values, such as 0.25, with high accuracy, but it cannot represent values ​​with large absolute values, such as 32 or more, and saturates. In the X2 layer, although the accuracy is lower than the X1 layer due to the larger quantization step, it can represent a wider range of values, so it can represent values ​​with large absolute values, such as 32 or more.

[0052] The activation unit 28 transforms each of the multiple shifted sum products with a different activation function. That is, the activation unit 28 uses a different activation function for each layer. Specifically, for shifted sum products with a relatively small quantization step, such as in the X1 layer, the activation unit 28 uses an activation function that represents a value with a relatively small absolute value. On the other hand, for shifted sum products with a relatively large quantization step, such as in the X2 layer, the activation unit 28 uses an activation function that represents a value with a relatively large absolute value.

[0053] Referring to Figure 6, an example of a method for determining different activation functions for each layer will be explained. In this example, similar to Figure 5, the quantization step in the X1 layer is 2 －２ Assume that the first feature map is obtained. In the X2 layer, the quantization step is 2０ The first feature map is obtained. The shifted sum of products is the sum of products after a right bit shift according to the shift amount (0 for layer X1, 2 for layer X2) determined by the range control unit 26. In reality, the output of each layer is scaled from the standpoint of computational efficiency, so in Figure 6, the shifted sum of products is shown as a value scaled to the equivalent of a signed 8-bit integer (an integer from -128 to 127).

[0054] The value range for the first feature map of layer X1 is -32 to 31.75, and the value range for the first feature map of layer X2 is -128 to 127. Therefore, in the first feature maps of layers X1 and X2, there may be duplicate values ​​output in the value range of -32 to 31.75.

[0055] The activation unit 28 determines an activation function for each layer that selectively selects the value of the first feature map output from any one of the layers for any overlapping value ranges. To do this, for example, an activation function for representing values ​​with relatively small absolute values, as shown in the upper part of Figure 6, and an activation function for representing values ​​with relatively large absolute values, as shown in the lower part of Figure 6, can be combined.

[0056] The activation unit 28 basically uses an activation function that selects the value of the first feature map with a relatively small quantization step for overlapping value ranges. This is based on the idea that a smaller quantization step allows for the representation of higher accuracy values, which is advantageous in reducing quantization errors.

[0057] Specifically, the activation unit 28 may determine a threshold for determining whether or not to represent a value with each activation function, according to the minimum value A and maximum value B that can be represented by the first feature map with a relatively small quantization step. For example, the activation unit 28 may use an activation function that represents a value greater than the minimum value A and less than the maximum value B as the activation function for selecting a value in the first feature map with a relatively small quantization step. Alternatively, the activation unit 28 may use an activation function that represents a value less than or equal to the minimum value A and greater than or equal to the maximum value B as the activation function for selecting a value in the first feature map with a relatively large quantization step.

[0058] Furthermore, it is preferable to express the minimum value A and maximum value B using an activation function applied to the shifted sum of products with a relatively large quantization step. This takes into account saturation in the first feature map with a relatively small quantization step. For example, in the X1 layer with a small quantization step, values ​​less than -32 and values ​​greater than 31.75 are subjected to saturation after scaling. That is, in the first feature map of the X1 layer, the scaled value of -128, which corresponds to the minimum value of the range, and the scaled value of 127, which corresponds to the maximum value of the range, may be the result of saturation.

[0059] Therefore, for example, in the X1 layer, the activation unit 28 may determine an activation function for the X1 layer such that the scaled values ​​output are between -127 and 126, and the values ​​of -128 and 127 are set to 0. This makes it possible to output highly accurate values ​​while avoiding outputting values ​​that may be saturated.

[0060] In the X2 layer, an activation function is determined that outputs 0 for values ​​within the range that can be output from the X1 layer. Due to the activation function of the X1 layer, values ​​in the range greater than -32 and less than 31.75 are output from the X1 layer, but values ​​less than or equal to -32 and greater than or equal to 31.75 are not output. Therefore, for example, the activation unit 28 determines an activation function as the activation function for the X2 layer that outputs values ​​less than or equal to -32 and greater than or equal to 31.75, and sets values ​​greater than -32 and less than 31.75 to 0. In this way, values ​​within the range that are not output by the X1 layer are output, while values ​​within the range that are not output by the X1 layer are output.

[0061] In this way, by determining the activation function so that the range of values ​​output from each layer does not overlap, the first feature map can be divided into multiple parts according to the magnitude of the values. That is, the activation function causes the values ​​of each grid cell in the first feature map to be activated in one layer, while the values ​​in the other layers are suppressed to 0. This makes it easy to accumulate the k first feature maps using linear operations in the Y layer described later.

[0062] In Figure 6, the activation function shows an example where the value of the first feature map of the X1 layer is selected across the entire range (excluding the maximum and minimum values) that can be represented by the first feature map of the X1 layer with a small quantization step. However, this is not the only example. The range in which a value with a small quantization step is selected can be arbitrarily determined within the range of values ​​that can be represented by the first feature map with a small quantization step.

[0063] Specifically, the activation unit 28 may determine the range of values ​​for which it selects values ​​for the first feature map with a small quantization step using an activation function, according to the distribution of sums of products. For example, if the distribution of values ​​is heavily biased and most values ​​can be represented without saturation even with the first feature map of the X1 layer which has a small quantization step, the absolute value of the boundary value set to 0 by the activation function of the X1 layer may be made smaller so that more values ​​are converted to 0. This reduces the number of values ​​that can be assigned to the first feature map of the X1 layer, thereby suppressing memory overflow 27. Memory overflow 27 can be detected by counting (monitoring) the maximum value that can be represented by the sum of products data type.

[0064] [Second Quantization Process] Next, the second quantization process in the Y layer will be explained. In the second quantization process, the multiplication unit 23 calculates the product of each of the multiple (k) first feature maps output from each of the X1, X2, ..., Xk layers with the kernel. The accumulation unit 24 calculates the sum of products in the channel direction and the layer direction from the multiple products. The sum of products in the layer direction refers to the sum of products of the Y1, Y2, ..., Yk layers.

[0065] However, as mentioned above, the first feature maps of layers X1, X2, ..., Xk have different quantization steps. In this case, depending on the quantization steps of the kernels in layers Y1, Y2, ..., Yk, the minimum step size for the product may differ for each layer, making it impossible to calculate the sum of products. Therefore, in layer Y, digit alignment in cumulative addition is achieved by pre-manipulating the bits of the kernels of each layer.

[0066] The control unit 30 bit-operations a kernel corresponding to at least one first feature map whose quantization step is not the smallest, based on the quantization step in each of the plurality (k) of first feature maps from the X layer.

[0067] Referring to FIG. 7, an example of the bit-operation method of the kernel will be described. FIG. 7 is a diagram showing the digit alignment of the product of the Y1 layer and the product of the Y2 layer. In FIG. 7, one bit is represented by one rectangle, and the numerical values from -4 to 7 in the uppermost row and the numerical values from -4 to 11 in the middle row indicate the bit weights. The first feature map and the kernel are represented by 8-bit fixed-point decimal numbers, and their product and sum of products are represented by 16-bit fixed-point decimal numbers. In the first feature maps of the X1 layer and the X2 layer, the quantization steps are different. The kernels of the Y1 layer and the Y2 layer originally have the same quantization step, which is 2 －２ Let's assume so.

[0068] The first feature map of the X1 layer input to the Y1 layer has a quantization step of 2 －２ Let's assume so. The kernel of the Y1 layer has a quantization step of 2 －２ Let's assume so. In this case, the minimum step width of the product of the first feature map and the kernel in the Y1 layer is 2 －４ Let's assume so.

[0069] The first feature map of the X2 layer input to the Y2 layer has a quantization step of 2 ０ Let's assume so. The kernel of the Y2 layer originally has a quantization step of 2 －２ Let's assume so. In this case, the minimum step width of the product of the first feature map and the kernel in the Y2 layer is 2 －２ Let's assume so. As a result, since the minimum step width of the product is different between the Y1 layer and the Y2 layer, adjustment is required for the calculation of the sum of products.

[0070] The control unit 30 bit-operations the kernels of each layer so that the minimum step width of the product of each layer matches the smallest minimum step width among the products of all layers. Specifically, since the quantization steps of the kernels are the same in each layer, the kernel is scaled by the difference in the quantization step of the first feature map.

[0071] For example, the quantization step of the first feature map of the X1 layer input to the Y1 layer is a minimum of 2 －２ Therefore, the quantization step of the first feature map of the X2 layer that is input to the Y2 layer to be bit-manipulated is 2 ０ The difference in quantization steps is 2. ２ Therefore, by discarding the upper 2 bits of the kernel in the Y2 layer and adding 2 bits with a value of 0 to the lower part, the minimum step size of the product in the Y2 layer is 2. －４ This allows for alignment of the digits of the product of Y1 layer and the product of Y2 layer, making addition possible.

[0072] As described above, the kernel is bit-manipulated for each layer. The multiplication unit 23 uses the bit-manipulated kernel to calculate the product of the first feature map and the kernel for each layer. That is, for at least one first feature map that does not have the minimum quantization step (for example, the first feature map of the X2 layer), the multiplication unit 23 calculates the product with the bit-manipulated kernel.

[0073] The cumulative unit 24 calculates (cumulatively adds) the sum of products in the channel direction and the layer direction. Specifically, the cumulative unit 24 calculates the sum of products for the input channels in layer Y1 and temporarily stores the calculated sum of products for the input channels in memory 27. Next, the cumulative unit 24 calculates the sum of products for the input channels in layer Y2 as well, then reads the sum of products for the input channels in layer Y1 stored in memory 27 and adds the sum of products for the input channels in layer Y2 to the read sum of products. This sum of products becomes the sum of products in the channel direction and the layer direction at layer Y2. The cumulative unit 24 repeats the cumulative addition in the channel direction and the layer direction up to layer Yk, and when the calculation of the sum of products up to layer Yk is complete, it outputs to the shift unit 25.

[0074] The shift unit 25 performs bit shifts on the sum of products in the channel direction and the layer direction. The amount of shift here may be predetermined for each convolutional layer, for example, or it may be appropriately determined by the range control unit 26 according to the range, bit width, quantization step, and computational efficiency of the next convolutional layer.

[0075] The activation unit 28 obtains an output feature map by transforming the sum of products in the channel direction and layer direction after bit shifting using a predetermined activation function. The activation function here may be one of several predetermined ones, such as ReLU, or it may be determined by the activation unit 28 as appropriate according to the range of the next convolutional layer, bit width, quantization step, and computational efficiency.

[0076] In other words, in the second quantization process, bit shifting, activation, and feature map output are not performed in the intermediate layers other than the final Yk layer. Also, memory 27 is not initialized in the intermediate layers other than the final Yk layer.

[0077] Furthermore, when performing bit manipulation on a kernel, the minimum step size of the kernel becomes smaller, thus narrowing the range of values ​​that the kernel can represent. Consequently, bit manipulation can result in the loss of information that was originally contained in the kernel, potentially leading to a decrease in computational precision.

[0078] Therefore, the control unit 30 may determine whether the precision of the product of the bit-manipulated kernel and the first feature map satisfies a predetermined criterion. For example, if the control unit 30 determines that a predetermined percentage (e.g., 50%) or more of the values ​​included in the original kernel are lost due to the bit manipulation, it may determine that the precision does not satisfy the predetermined criterion.

[0079] Furthermore, if the control unit 30 determines that the criteria are not met, it may cancel the bit manipulation of the kernel. In this case, it may adjust the quantization step of the first feature map corresponding to the kernel. For example, in Figure 7, the quantization step of the Y2 layer kernel is 2 －２ The quantization step of the first feature map of the X2 layer, which is input to the Y2 layer, is 2 －２ To achieve this, a right bit shift with a shift amount of 2 may be performed.

[0080] Furthermore, if the control unit 30 determines that the criteria are not met, it may expand the bit width of the bit-manipulated kernel. That is, an 8-bit fixed-point kernel may be made 9 bits or more.

[0081] In the above explanation, the number of divisions k was assumed to be 2, but the number of divisions k may be 3 or more. In this case as well, the shift amount and activation function can be determined for each layer.

[0082] Next, the operation of the information processing device 10 will be described. Figure 8 is a flowchart showing the flow of information processing by the information processing device 10. Information processing is the processing from the input to the output of a feature map in one convolutional layer. Figure 9 is a flowchart showing the flow of the first quantization processing included in the information processing. Figure 10 is a flowchart showing the flow of the second quantization processing included in the information processing. The CPU 11 reads the information processing program from the ROM 12 or storage 14, expands it in the RAM 13 and executes it, thereby performing information processing, the first quantization processing, and the second quantization processing. Note that the information processing is an example of the information processing method of this disclosure.

[0083] In step S10, the CPU 11, acting as the control unit 30, acquires the input feature map and kernel for the convolutional layer and inputs them to the CNN dedicated circuit 20. In step S20, the CPU 11 and the CNN dedicated circuit 20 execute the first quantization process. The first quantization process will now be described with reference to Figure 9. As described above, the first quantization process is performed in the X layer of the k-layer structure included in the convolutional layer.

[0084] In step S22, the CNN-dedicated circuit 20, as a range control unit 26, determines the shift amount for the bit shift in step S30 described later, for each layer X1, X2, ..., Xk. The shift amount can be determined, for example, according to the range of the next convolutional layer and the quantization step of each layer. In step S24, the CNN-dedicated circuit 20, as an activation unit 28, determines the activation function to be used in step S32 described later, for each layer. For example, in layers with a relatively small quantization step, an activation function that represents a value with a relatively small absolute value may be used, and in layers with a relatively large quantization step, an activation function that represents a value with a relatively large absolute value may be used.

[0085] The CNN-dedicated circuit 20 first executes the processing from step S26 to step S38 for the X1 layer. In step S26, the CNN-dedicated circuit 20, as a multiplication unit 23, calculates (multiplies) the product of the input feature map and kernel input in step S10. If the input feature map and kernel have multiple input channels, the product is calculated for each input channel. In step S28, the CNN-dedicated circuit 20, as an accumulation unit 24, calculates (cumulatively adds) the sum of products in the channel direction using the product calculated in step S26. The calculation of the sum of products can be performed, for example, by repeatedly updating the sum of products for each input channel while temporarily storing the value in memory 27.

[0086] In step S30, the CNN-dedicated circuit 20, acting as a shift unit 25, right-bits the sum of products calculated in step S28 according to the shift amount determined in step S22. In step S32, the CNN-dedicated circuit 20, acting as an activation unit 28, obtains a first feature map by nonlinearly transforming the sum of products after the shift in step S30 using the activation function determined in step S24.

[0087] In step S34, the CNN-dedicated circuit 20, acting as an activation unit 28, outputs the first feature map obtained in step S32 to the control unit 30. The CPU 11, acting as the control unit 30, stores the acquired first feature map in a storage medium such as ROM 12 so that it can be used in step S14, described later. In step S36, the CNN-dedicated circuit 20, acting as an accumulation unit 24, initializes the sum of products temporarily stored in memory 27. In step S38, the control unit 30 determines whether the output of the first feature map for k layers is complete. If not, it returns to step S26, and the CNN-dedicated circuit 20 executes the processing from step S26 to step S38 for the next layer (for example, layer X2). That is, the processing from step S26 to step S38 is repeated for k layers, and when completed, the first quantization process is terminated.

[0088] Returning to Figure 8, in step S12, the CPU 11, as the control unit 30, identifies the kernel to be bit-manipulated from the kernels acquired in step S10 and performs bit manipulation. The kernel to be bit-manipulated is, for example, the kernel corresponding to at least one first feature map among the k-layer first feature maps obtained by the first quantization process in step S20 that does not have the minimum quantization step. In step S14, the CPU 11, as the control unit 30, inputs the k-layer first feature maps obtained by the first quantization process in step S20 and the kernels to the CNN dedicated circuit 20. Of the input kernels, the kernel to be bit-manipulated is the one bit-manipulated in step S12, and the kernel not to be bit-manipulated is the one acquired in step S10.

[0089] In step S40, the CPU 11 and the CNN dedicated circuit 20 perform the second quantization process. The second quantization process will now be explained with reference to Figure 10. As described above, the second quantization process is performed in the Y layer of the k-layer structure included in the convolutional layer. The CNN dedicated circuit 20 first performs the processes from step S42 to step S46 on the Y1 layer.

[0090] In step S42, the CNN-dedicated circuit 20, acting as a multiplication unit 23, calculates (multiplies) the product of the first feature map and the kernel input in step S14. If the first feature map and kernel have multiple input channels, the product is calculated for each input channel. In step S44, the CNN-dedicated circuit 20, acting as an accumulation unit 24, calculates (cumulatively adds) the sum of products in the channel direction and the layer direction using the product calculated in step S42. The calculation of the sum of products can be performed, for example, by repeatedly updating the sum of products for each input channel and for k layers while temporarily storing the values ​​in memory 27.

[0091] In step S46, the CNN-dedicated circuit 20, acting as the cumulative unit 24, determines whether the cumulative addition for k layers has been completed. If not, it returns to step S42, and the CNN-dedicated circuit 20 executes the processes from step S42 to step S46 for the next layer (for example, layer Y2). That is, the processes from step S42 to step S46 are repeated for k layers, and if completed, it proceeds to step S48.

[0092] In step S48, the CNN-dedicated circuit 20, acting as a shift unit 25, right-bits the sum of products in the channel direction and layer direction calculated in step S44. In step S50, the CNN-dedicated circuit 20, acting as an activation unit 28, obtains an output feature map by nonlinearly transforming the sum of products after the shift in step S48 using an activation function.

[0093] In step S52, the CNN-dedicated circuit 20, acting as an activation unit 28, outputs the output feature map obtained in step S50 to the control unit 30. In step S54, the CNN-dedicated circuit 20, acting as an accumulation unit 24, initializes the sum of products temporarily stored in the memory 27 and terminates the second quantization process.

[0094] Returning to Figure 8, in step S16, the CPU 11, acting as the control unit 30, stores the output feature map obtained by the second quantization process in step S40 in a storage medium such as ROM 12, so that it can be used as the input feature map for the next convolutional layer. Once step S16 is completed, this information processing is terminated.

[0095] In the first quantization process, the input feature maps and kernels input to each layer X1, X2, ..., Xk are common, so the sum of products will be the same value in each layer. Therefore, the accumulation unit 24 may, for example, temporarily store the sum of products calculated in the first X1 layer in memory 27, and reuse it by reading the sum of products from memory 27 in the X2 layer and beyond. In this case, the calculation of the product by the multiplication unit 23 and the calculation of the sum of products by the accumulation unit 24 in the X2 layer and beyond can be omitted. That is, by changing the return destination of step S38 from before step S26 to before step S30, the processing from step S26 to step S28 may be performed only in the first X1 layer. In this case, the initialization of memory 27 in step S36 should be performed after step S38.

[0096] As described above, the information processing device 10 according to this embodiment is an information processing device using a convolutional neural network, and comprises a multiplication unit 23 that calculates the product of an input feature map and a kernel, an accumulation unit 24 that calculates the sum of products from the product, a shift unit 25 that bit-shifts the sum of products, and an activation unit 28 that converts the sum of products after bit shifting using an activation function.

[0097] The information processing device 10 performs a first quantization process which includes the following: The shift unit 25 obtains multiple shifted sums of products by bit-shifting the sum of products multiple times (k times) while changing the shift amount. The activation unit 28 uses an activation function that represents a value with a relatively small absolute value for shifted sums of products with a relatively small quantization step, and uses an activation function that represents a value with a relatively large absolute value for shifted sums of products with a relatively large quantization step. By performing the first quantization process, multiple first feature maps are obtained.

[0098] The first quantization process allows a single input feature map to be divided into multiple first feature maps with different quantization steps. Each first feature map, viewed individually, has a single fixed quantization step, enabling calculations using a simple CNN-dedicated circuit 20, similar to the case where one quantization step is set per layer, thus reducing circuit area. On the other hand, by considering the multiple first feature maps collectively, multiple quantization steps can be set per layer, thus reducing quantization errors. In other words, the information processing device 10 according to this embodiment achieves both circuit area reduction and quantization error reduction.

[0099] In the above embodiment, a configuration was described in which the first quantization process and the second quantization process are performed once each in a single convolutional layer, but the invention is not limited to this configuration. The combination of one first quantization process and one second quantization process may be used as a processing unit, and the first quantization process and the second quantization process may be repeatedly performed in a single convolutional layer.

[0100] For example, one convolutional layer is k １ Layered structure Xa layer, k １ Layered Ya layer, k ２ Layered structure Xb layer and k ２ The layered structure of the Yb layer may be broadly represented as a four-layer structure. Number of divisions: k １ and k ２ k is an integer greater than or equal to 2, and may be the same value or different values. The Xa layer uses a first quantization process to obtain k from the input feature map to this convolutional layer. １ It outputs a first feature map of a certain number. The Ya layer is subjected to a second quantization process, and the k of the Xa layer １ The Xb layer outputs one feature map from the first feature maps. The Xb layer performs a first quantization process, and the k feature maps of the Ya layer are converted to k ２ It outputs a first feature map of a certain number. The Yb layer undergoes a second quantization process, and the k of the Xb layer ２ One feature map is output from each of the first feature maps. The feature map of the Yb layer becomes the output feature map of this convolutional layer.

[0101] Furthermore, in the above embodiment, the kernels used in the first quantization process and the second quantization process may be the same or different. For example, the kernel used in the X1 layer and the kernel used in the Y1 layer before bit manipulation may be the same or different. Also, for example, in the above example of a four-layer structure, the kernel used in the Xa layer and the kernel used in the Xb layer may be the same or different.

[0102] Furthermore, in the above embodiment, the processing of the control unit 30 was described as being performed by the CPU 11, and the processing of the sum-accumulate unit 22, the range control unit 26, and the activation unit 28 was described as being performed by the CNN dedicated circuit 20, but this is not limited to this. For example, the CNN dedicated circuit 20 may perform part or all of the processing of the control unit 30. Alternatively, for example, the CPU 11 may perform part or all of the processing of the sum-accumulate unit 22, the range control unit 26, and the activation unit 28.

[0103] Furthermore, the information processing that the CPU reads and executes in the above embodiment may be executed by various processors other than the CPU. Examples of such processors include PLDs whose circuit configuration can be changed after manufacturing, such as FPGAs, and dedicated electrical circuits which are processors with circuit configurations specifically designed to execute particular processing, such as ASICs. The information processing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0104] Furthermore, although the above embodiment describes an embodiment in which the information processing program is pre-stored (installed) in ROM 12 or storage 14, the invention is not limited to this. The program may be provided in a form stored on a non-transitor storage medium such as CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), and USB (Universal Serial Bus) memory. Alternatively, the program may be provided in a form downloaded from an external device via a network.

[0105] The following additional information is disclosed regarding the embodiments described above.

[0106] (Note 1) An information processing device using a convolutional neural network, comprising: a memory; and at least one processor connected to the memory, wherein the processor includes: calculating the product of an input feature map and a kernel; calculating the sum of products from the product; bit-shifting the sum of products; and transforming the bit-shifted sum of products using an activation function, wherein the bit-shifting involves bit-shifting the sum of products multiple times while changing the shift amount to obtain multiple shifted sums of products; and the transformation involves performing a first quantization process that includes using an activation function that represents a value with a relatively large absolute value for the shifted sums of products with a relatively large quantization step, and using an activation function that represents a value with a relatively small absolute value for the shifted sums of products with a relatively small quantization step to obtain multiple first feature maps.

[0107] (Appendix 2) A non-temporary storage medium storing a program executable by a computer to perform information processing by a convolutional neural network, wherein the information processing includes: calculating the product of an input feature map and a kernel; calculating the sum of products from the product; bit-shifting the sum of products; and transforming the bit-shifted sum of products using an activation function, wherein the bit-shifting involves bit-shifting the sum of products multiple times while changing the shift amount to obtain multiple shifted sums of products; and the transformation involves performing a first quantization process to obtain multiple first feature maps, wherein the activation function used for the shifted sums of products with a relatively large quantization step represents a value with a relatively large absolute value, and the activation function used for the shifted sums of products with a relatively small quantization step represents a value with a relatively small absolute value.

[0108] 10 Information Processing Device 11 CPU 12 ROM 13 RAM 14 Storage 15 Input Unit 16 Display Unit 17 Communication Interface 19 Bus 20 CNN Dedicated Circuit 22 Multiply-Accumulate Unit 23 Multiplication Unit 24 Accumulation Unit 25 Shift Unit 26 Range Control Unit 27 Memory 28 Activation Unit 30 Control Unit

Claims

1. An information processing device using a convolutional neural network, comprising: a multiplication unit that calculates the product of an input feature map and a kernel; an accumulation unit that calculates the sum of products from the product; a shift unit that bit-shifts the sum of products; and an activation unit that transforms the bit-shifted sum of products using an activation function, wherein the shift unit obtains a plurality of shifted sums of products by bit-shifting the sum of products multiple times while changing the shift amount; and the activation unit performs a first quantization process that includes using an activation function that represents a value with a relatively small absolute value for the shifted sums of products with a relatively small quantization step, and using an activation function that represents a value with a relatively large absolute value for the shifted sums of products with a relatively large quantization step, thereby obtaining a plurality of first feature maps.

2. An information processing apparatus according to claim 1, further comprising a control unit for bit-manipulating a kernel corresponding to at least one of the first feature maps that does not have the minimum quantization step, wherein the multiplication unit calculates the product of each of the plurality of first feature maps with the kernel, and calculates the product of at least one of the first feature maps that does not have the minimum quantization step with the bit-manipulated kernel, the accumulation unit calculates a sum of products from the plurality of products, the shift unit bit-shifts the sum of products, and the activation unit performs a second quantization process to obtain an output feature map, the second quantization process being performed by converting the sum of products after the bit shift using a predetermined activation function.

3. The information processing apparatus according to claim 2, wherein the control unit determines whether the precision of the product of the bit-manipulated kernel and the first feature map satisfies a predetermined criterion, and if the criterion is not met, the bit manipulation of the kernel is stopped or the bit width of the bit-manipulated kernel is expanded.

4. An information processing method using a convolutional neural network, comprising: calculating the product of an input feature map and a kernel; calculating the sum of products from the product; bit-shifting the sum of products; and transforming the bit-shifted sum of products using an activation function, wherein the bit-shifting involves bit-shifting the sum of products multiple times while varying the shift amount to obtain multiple shifted sums of products; and the transformation involves using an activation function that represents a value with a relatively small absolute value for the shifted sums of products with a relatively small quantization step, and using an activation function that represents a value with a relatively large absolute value for the shifted sums of products with a relatively large quantization step, thereby obtaining multiple first feature maps, by performing a first quantization process.

Images