Computation device and computation method
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- NTT INNOVATIVE DEVICES CORP
- Filing Date
- 2026-01-26
- Publication Date
- 2026-08-06
Smart Images
Figure JP2026002457_06082026_PF_FP_ABST
Abstract
Description
Computing device and computing method
[0001] The present disclosure relates to a computing device and a computing method.
[0002] In an LSI such as an image encoding LSI, a small-capacity temporary recording device called a cache memory that can operate at high speed is mounted inside the LSI in order to bridge the performance gap between the LSI and an external main memory (Patent Document 1).
[0003] Japanese Patent Application Laid-Open No. 2022-21062
[0004] When performing a convolution process used in object detection processing or the like on an LSI, input of data called image data, data called a kernel (or weight) (hereinafter referred to as kernel data), and data called a bias is required. For example, in the case of YOLOv3, the maximum size of the input image data is 608×608 pixels (a total of 369,664 bytes when 1 pixel is 1 byte), and the total size of the kernel data is approximately 250,000,000 bytes when the accuracy of the kernel data is 32 bits.
[0005] When holding kernel data in the memory inside the LSI, the area of that memory becomes very large, resulting in an extremely low manufacturing yield of the LSI and an extremely high unit cost per chip. Therefore, it is desirable to read the kernel data from an external memory and perform convolution processing.
[0006] On the other hand, in YOL0v3 or the like, convolution processing is performed using the same kernel data multiple times. However, if the same kernel data is read from an external main memory multiple times, there is a problem that wasted power consumption occurs and performance deteriorates when a cache miss occurs.
[0007] The present disclosure has been made in view of the above, and an object thereof is to provide a device that speeds up convolution processing.
[0008] An arithmetic unit according to one aspect of the present disclosure is an arithmetic unit that performs image convolution processing using kernel data, comprising: a plurality of kernel memories for holding the kernel data; an image memory for holding image data necessary for at least one convolution process; a kernel data writing unit for reading the kernel data necessary for the convolution process from an external memory and writing it to the kernel memory; an image data writing unit for reading image data necessary for at least one convolution process from an external memory and writing it to the image memory; a kernel data output unit for reading and outputting the kernel data from the kernel memory; an image data output unit for reading and outputting the image data from the image memory; and an arithmetic unit for performing convolution processing using the kernel data and the image data, wherein the kernel data writing unit writes kernel data to be used for different convolution processes to one of the plurality of kernel memories, and the kernel data output unit switches the kernel memory from which to read the kernel data according to the convolution process.
[0009] According to this disclosure, a device that speeds up convolution processing can be provided.
[0010] Figure 1 shows an example of the configuration of the arithmetic unit of the first embodiment. Figure 2 is a sequence diagram showing an example of the processing flow of each part of the arithmetic unit of the first embodiment. Figure 3 shows an example of the configuration of the arithmetic unit of the second embodiment. Figure 4 is a sequence diagram showing an example of the processing flow of each part of the arithmetic unit of the second embodiment. Figure 5 shows an example of the configuration of the kernel data output unit. Figure 6 shows an example of writing to 32 memory and pixels. Figure 7 shows an example of writing to pixels when the number of simultaneous operations is 4. Figure 8 shows an example of writing to kernel memory in mode 1. Figure 9 shows an example of writing to kernel memory in mode 2.
[0011] [First Embodiment] Referring to Figure 1, an example of the configuration of the arithmetic unit of the first embodiment will be described. The arithmetic unit shown in the figure is an LSI that performs YOLOv3 processing. In YOLOv3 processing, a 105-layer neural network is used to perform convolution processing using image data and kernel data. Although bias data is also used in the convolution processing, the explanation of bias data will be omitted. In addition to convolution processing, YOLOv3 processing also performs activation processing, etc., but the explanation will be omitted.
[0012] The arithmetic unit shown in Figure 1 comprises a kernel data writing unit 11, an image data writing unit 12, a kernel data output unit 13, an image data output unit 14, an arithmetic unit 15, a first kernel memory 21, a second kernel memory 22, and an image memory 23.
[0013] The kernel data writing unit 11 reads the kernel data for each layer of the neural network from external memory and writes it to kernel memories 21 and 22. Specifically, the kernel data writing unit 11 writes the kernel data for the next layer to be processed alternately to the first kernel memory 21 and the second kernel memory 22 for each layer.
[0014] The image data writing unit 12 reads a portion of the image data (the original image data in layer 0) that has undergone convolution processing, activation processing, etc., in the previous layer (at least one convolution processing step, for example, 32x32 pixels) from external memory and writes it to the image memory 23. When the convolution processing of the portion of the image data written to the image memory 23 is completed, the image data writing unit 12 reads the next portion of the image data from external memory and writes it to the image memory 23. Hereinafter, the portion of the image data written to the image memory 23 may simply be referred to as image data.
[0015] The kernel data output unit 13 reads kernel data from kernel memories 21 and 22, which hold the kernel data of the currently processed layer, and outputs it to the arithmetic unit 15. In other words, the kernel data output unit 13 alternately reads kernel data from either the first kernel memory 21 or the second kernel memory 22 for each layer and outputs it to the arithmetic unit 15.
[0016] The image data output unit 14 reads image data from the image memory 23 and outputs it to the calculation unit 15.
[0017] The arithmetic unit 15 executes convolution processing using the image data and kernel data input from the image data output unit 14 and the kernel data output unit 13. The convolution processing also uses bias data input from a means for outputting bias data (not shown). The result of the convolution processing by the arithmetic unit 15 is used as the image data for the next layer after activation processing and other operations are performed. If the arithmetic unit 15 performs all necessary processing such as activation processing, the processing result may be written to external memory by the arithmetic unit 15. If activation processing and other operations are performed by a circuit other than the arithmetic unit 15, the necessary processing such as activation processing is performed on the output of the arithmetic unit 15 and written to external memory. The image data writing unit 12 reads the image data from that external memory. The result of the convolution processing of the last layer is output as the YOLOv3 processing result after activation processing and other operations are performed.
[0018] The kernel memories 21 and 22 are dedicated memories that hold kernel data for each layer.
[0019] The image memory 23 is a dedicated memory that holds a portion of the image data processed in each layer.
[0020] The kernel data writing unit 11, image data writing unit 12, kernel data output unit 13, image data output unit 14, and arithmetic unit 15 described above may be dedicated circuits, or the CPU may perform the processing of each unit according to the program.
[0021] Referring to the sequence diagram in Figure 2, an example of the processing flow of each part of the arithmetic unit will be explained. In the same diagram, processes that are executed in parallel are shown at the same height. Although bias data is also used in the convolution process, the explanation of bias data will be omitted. Furthermore, explanations of activation processes, etc., will also be omitted.
[0022] In step S101, the kernel data writing unit 11 reads the kernel data for layer 0 from external memory and writes it to the first kernel memory 21.
[0023] In step S102, the image data writing unit 12 reads a portion of the 0-layer image data from the external memory and writes it to the image memory 23.
[0024] After the kernel data and image data write processing (steps S101, S102) is completed, in step S111, the kernel data output unit 13 reads the kernel data of the 0th layer from the first kernel memory 21 and outputs it to the calculation unit 15. In step S112, the image data output unit 14 reads a portion of the image data of the 0th layer from the image memory 23 and outputs it to the calculation unit 15. In step S113, the calculation unit 15 performs convolution processing using the image data and kernel data.
[0025] In parallel with the processing in steps S111 to S113, in step S114, the kernel data writing unit 11 reads the kernel data for layer 1, which will be processed after layer 0, from external memory and writes it to the second kernel memory 22. The processing in step S114 is carried out continuously without synchronization with the other parts.
[0026] After the convolution process on the data in the image memory 23 (step S113) is completed, in step S121, the image data writing unit 12 reads a portion of the image data to be processed next from the external memory and writes it to the image memory 23. As for the kernel data, since the data stored in the first kernel memory 21 is used, it is not necessary for the kernel data writing unit 11 to write the kernel data of layer 0 to the first kernel memory 21.
[0027] After the image data writing unit 12 completes the image data writing process (step S121), in step S131, the kernel data output unit 13 reads the kernel data of the 0th layer from the first kernel memory 21 and outputs it to the arithmetic unit 15. In step S132, the image data output unit 14 reads a portion of the image data of the 0th layer from the image memory 23 and outputs it to the arithmetic unit 15. In step S133, the arithmetic unit 15 performs convolution processing using the image data and kernel data.
[0028] The process of rewriting the image memory 23 in step S121 and repeating the convolution processes in steps S131, S132, and S133 is repeated until the convolution process on the entire image data of layer 0 is completed.
[0029] After the convolution process on the entire image data of layer 0 is completed, in step S201, the image data writing unit 12 reads a portion of the image data of layer 1 from the external memory and writes it to the image memory 23.
[0030] After the writing process of the first layer kernel data and image data (steps S114, S201) is completed, in step S211, the kernel data output unit 13 reads the first layer kernel data from the second kernel memory 22 and outputs it to the calculation unit 15, in step S212, the image data output unit 14 reads a portion of the first layer image data from the image memory 23 and outputs it to the calculation unit 15, and in step S213, the calculation unit 15 performs convolution processing using the image data and kernel data.
[0031] After the convolution processing on the 0-layer image data is completed, in parallel with the above processing, in step S202, the kernel data writing unit 11 reads the 2-layer kernel data from external memory and writes it to the first kernel memory 21. The processing in step S202 is carried out continuously without synchronization with the other units.
[0032] Similarly to layer 0, layer 1 also repeats the process of rewriting the image memory 23 and performing convolution.
[0033] After the convolution process on the entire image data of the first layer is completed, for each layer of YOKOv3, the kernel memory 21 and 22 are used alternately, similar to the process for layer 0 and layer 1, to rewrite the image memory 23 and perform the convolution process.
[0034] Note that the image data for the 0th layer of YOLOv3 is 608 x 608 pixels, and if the capacity of the image memory 23 is 32 x 32 pixels, then (608 x 608) / (32 x 32) = 361. Therefore, the image memory 23 will be rewritten 361 times (including the first time) before the convolution processing of the 0th layer image data is completed. Rewriting the 0th layer kernel data is not necessary when rewriting the image memory 23.
[0035] Furthermore, if the entire image data of layer 0 contains multiple 608x608 pixel image data, for example, if there are 8 608x608 pixel image data in layer 0, the arithmetic unit will perform convolution processing for 8 608x608 pixels consecutively, resulting in 361 x 8 = 2,888 rewrites of the image memory 23. In this case, rewriting of the layer 0 kernel data is unnecessary until all 8 608x608 pixel convolution processes are completed.
[0036] Regarding the capacity of kernel memory 21 and 22, in YOLOv3, if the kernel data precision is 32 bits, the maximum size of kernel data for one layer is 18,874,368 bytes. Therefore, if the kernel memory 21 and 22 have a capacity greater than this, kernel data will only be written once per layer. Furthermore, if the image data is 8 bits / pixel, and the kernel data precision is also set to 8 bits, the required capacity of kernel memory will be reduced to 1 / 4 of the above.
[0037] In the configuration of the arithmetic unit according to the first embodiment, without using a so-called cache memory, the reads from the kernel memories 21 and 22 and the image memory 23 always have the same performance, and there is no performance degradation due to cache misses. If SRAMs with a width of 1024 bits are used as the kernel memories 21 and 22 and the image memory 23, the performance for both read and write is 128 Byte / clk, and performance equivalent to or higher than that of access using a cache memory can be obtained.
[0038] Also, by alternately using the first kernel memory 21 and the second kernel memory 22 for each layer, for example, it is possible to simultaneously perform the convolution processing for layer 0 and the write of kernel data for layer 1. When there is only one kernel memory, after the convolution processing for layer 0 is completed, the write of kernel data for layer 1 is performed, so the convolution processing for layer 1 cannot be started until the write of the necessary data is completed. In a configuration with two kernel memories, if the write of kernel data for layer 1 is completed before the convolution processing for layer 0 is completed, the convolution processing for layer 1 can be started immediately when the convolution processing for layer 0 is completed. As a result, the waiting time is eliminated or reduced, so the performance of the convolution processing is improved.
[0039] [Second Embodiment] Referring to FIG. 3, an example of the configuration of the arithmetic unit according to the second embodiment will be described. The arithmetic unit in FIG. 3 is different from the first embodiment in that it includes a plurality of image memories 23 and 24. Redundant descriptions of the same configuration as in the first embodiment will be omitted.
[0040] The image data writing unit 12 reads a part of the image data from the external memory and alternately writes the image data to the image memories 23 and 24 for each convolution processing.
[0041] The image data output unit 14 alternately reads the image data from the image memories 23 and 24 and outputs it to the arithmetic unit 15.
[0042] Referring to the sequence diagram of FIG. 4, an example of the processing flow of each part of the arithmetic unit will be described. In the figure, the processes executed in parallel are set at the same height. Although bias data is also used in the convolution process, the description of the bias data is omitted. Also, the description of the activation process and the like is omitted.
[0043] In step S301, the kernel data writing unit 11 reads the kernel data of layer 0 from the external memory and writes it to the first kernel memory 21.
[0044] In step S302, the image data writing unit 12 reads a part of the image data of layer 0 from the external memory and writes it to the first image memory 23.
[0045] After the writing processes of the kernel data and the image data (steps S301 and S302) are completed, in step S311, the kernel data output unit 13 reads the kernel data of layer 0 from the first kernel memory 21 and outputs it to the arithmetic unit 15. In step S312, the image data output unit 14 reads a part of the image data of layer 0 from the first image memory 23 and outputs it to the arithmetic unit 15. In step S313, the arithmetic unit 15 executes a convolution process using the image data and the kernel data.
[0046] In parallel with the processes of steps S311 to S313, in step S314, the kernel data writing unit 11 reads the kernel data of layer 1 to be processed next after layer 0 from the external memory and writes it to the second kernel memory 22. The process of step S314 is continuously performed without synchronization with other parts. Also, in step S315, the image data writing unit 12 reads the image data to be processed next from the external memory and writes it to the second image memory 24.
[0047] After the convolution process on the data in the first image memory 23 (step S313) is completed, in step S324, the image data writing unit 12 reads the image data to be processed the next time (third time) from the external memory and writes it to the first image memory 23. As for the kernel data, since the data stored in the first kernel memory 21 is used, it is not necessary for the kernel data writing unit 11 to write the kernel data for layer 0 to the first kernel memory 21.
[0048] After the image data writing unit 12 completes the writing process of image data to the second image memory 24 (step S315), in step S321, the kernel data output unit 13 reads the kernel data of the 0th layer from the first kernel memory 21 and outputs it to the calculation unit 15. In step S322, the image data output unit 14 reads a portion of the image data of the 0th layer from the second image memory 24 and outputs it to the calculation unit 15. In step S323, the calculation unit 15 performs convolution processing using the image data and kernel data.
[0049] After the convolution process on the data in the second image memory 24 (step S323) is completed, in step S334, the image data writing unit 12 reads the image data to be processed the next time (fourth time) from the external memory and writes it to the second image memory 24. As for the kernel data, the data stored in the first kernel memory 21 is used, so it is not necessary for the kernel data writing unit 11 to write the kernel data for layer 0 to the first kernel memory 21.
[0050] After the image data writing unit 12 completes the writing process of image data to the first image memory 23 (step S324), in step S331, the kernel data output unit 13 reads the kernel data of the 0th layer from the first kernel memory 21 and outputs it to the calculation unit 15. In step S332, the image data output unit 14 reads a portion of the image data of the 0th layer from the first image memory 23 and outputs it to the calculation unit 15. In step S333, the calculation unit 15 performs convolution processing using the image data and kernel data.
[0051] The following steps, S324 and S334, which involve rewriting the image memories 23 and 24, and S321 to S323 and S331 to S333, are repeated until the convolution process for the entire image data of layer 0 is completed.
[0052] After the convolution process on the entire image data of layer 0 is completed, in step S401, the image data writing unit 12 reads a portion of the image data of layer 1 from the external memory and writes it to the first image memory 23.
[0053] After the writing process of the first layer kernel data and image data (steps S314, S401) is completed, in step S411, the kernel data output unit 13 reads the first layer kernel data from the second kernel memory 22 and outputs it to the calculation unit 15, in step S412, the image data output unit 14 reads a portion of the first layer image data from the first image memory 23 and outputs it to the calculation unit 15, and in step S413, the calculation unit 15 performs convolution processing using the image data and kernel data.
[0054] After the convolution processing on the 0-layer image data is completed, in parallel with the above processing, in step S402, the kernel data writing unit 11 reads the 2-layer kernel data from external memory and writes it to the first kernel memory 21. The processing in step S402 is carried out continuously without synchronization with the other units. Also, in step S414, the image data writing unit 12 reads the image data to be processed next from external memory and writes it to the second image memory 24.
[0055] Similarly to layer 0, layer 1 also repeats the process of rewriting image memories 23 and 24 and performing convolution.
[0056] After the convolution process on the entire image data of layer 1 is completed, for each layer of YOKOv3, the kernel memory 21 and 22 are used alternately, similar to layer 0 and layer 1, to rewrite the image memory 23 and 24 and perform the convolution process.
[0057] Note that the image data for the 0th layer of YOLOv3 is 608 x 608 pixels, and assuming that the capacity of image memory 23 and 24 is 32 x 32 pixels each, then (608 x 608) / (32 x 32) = 361. Therefore, the first image memory 23 will be rewritten 181 times (including the first rewrite) and the second image memory 24 will be rewritten 180 times (including the first rewrite) before all the convolution processing of the 0th layer image data is completed. Rewriting the 0th layer kernel data is not necessary when rewriting image memory 23 and 24.
[0058] Furthermore, if there are multiple 608x608 pixel image data in the entire 0-layer image data, for example, if there are eight 608x608 pixel image data in the 0-layer, the arithmetic unit will perform convolution processing for eight 608x608 pixels consecutively. This will result in 181 x 8 = 1,448 rewrites of the first image memory 23 and 180 x 8 = 1,440 rewrites of the second image memory 24. In this case, rewriting of the 0-layer kernel data is unnecessary until all 608x608 pixel x 8 convolution processing is completed.
[0059] Regarding the capacity of kernel memory 21 and 22, in YOLOv3, if the kernel data precision is 32 bits, the maximum size of kernel data for one layer is 18,874,368 bytes. Therefore, if the kernel memory 21 and 22 have a capacity greater than this, kernel data will only be written once per layer. Furthermore, if the image data is 8 bits / pixel, and the kernel data precision is also set to 8 bits, the required capacity of kernel memory will be reduced to 1 / 4 of the above.
[0060] In the configuration of the arithmetic unit of the second embodiment, so-called cache memory is not used, and reads from kernel memories 21, 22 and image memories 23, 24 always have the same performance, and no performance degradation due to cache misses occurs. If, for example, 1024-bit wide SRAM is used as kernel memories 21, 22 and image memories 23, 24, both read and write performance will be 128 bytes / clk, which is equivalent to or better than access using cache memory.
[0061] Furthermore, by using the first kernel memory 21 and the second kernel memory 22 alternately for each layer, it is possible to perform, for example, the convolution process of layer 0 and the writing of kernel data for layer 1 simultaneously. However, if there is only one kernel memory, the writing of kernel data for layer 1 is performed after the convolution process of layer 0 is completed, so the convolution process of layer 1 cannot start until the writing of the necessary data is complete. In a configuration with two kernel memories, if the writing of kernel data for layer 1 is completed before the convolution process of layer 0 is completed, the convolution process of layer 1 can start immediately after the convolution process of layer 0 is completed. As a result, the waiting time is eliminated or reduced, improving the performance of the convolution process.
[0062] Furthermore, by using the first image memory 23 and the second image memory 24 alternately, for example, the convolution process of the first image data of layer 0 and the write process of the second image data of layer 0 can be performed simultaneously. Note that if there is only one image memory, the write process of the second image data of layer 0 is performed after the convolution process of the first image data of layer 0 is completed, so the next convolution process cannot start until the write of the necessary data is complete. In a configuration with two image memories, even when using so-called single-port SRAM, if the write process of the second image data of layer 0 is completed before the convolution process of the first image data of layer 0 is completed, the convolution process of the second image data of layer 0 can start immediately after the convolution process of the first image data of layer 0 is completed. As a result, the waiting time is eliminated or reduced, improving the performance of the convolution process. Note that if dual-port SRAM is used instead of single-port SRAM, it is possible to write and read to the same SRAM simultaneously, but when doing so, it is necessary to select only rewritable addresses for writing.
[0063] [Third Embodiment] The third embodiment supports three types of convolution processing: 3x3 convolution (stride=1), 1x1 convolution (stride=1), and 3x3 convolution (stride=2), using the same image memory and the same kernel memory. Hereinafter, the output modes of the kernel data output unit 13 and the image data output unit 14 in each convolution processing will be referred to as Mode 1, Mode 2, and Mode 3.
[0064] The output modes of the kernel data output unit 13 for each of the above convolution processes are shown below.
[0065] Mode 1: 32 bits × 9 × number of simultaneous operations (ich) × number of simultaneous operations (och) × 1 time Mode 2: 32 bits × 1 × number of simultaneous operations (ich) × number of simultaneous operations (och) × 1 time Mode 3: 32 bits × 1 × number of simultaneous operations (ich) × number of simultaneous operations (och) × 9 times
[0066] ich (input channels) refers to the number of data channels used as input for the convolution operation. For example, a color image typically has three channels (R, G, B), so the number of input channels is 3. och (output channels) refers to the number of feature maps generated as a result of the convolution operation. For example, if 32 filters are used in a layer, the number of output channels will be 32.
[0067] Figure 5 shows an example of the configuration of the kernel data output unit 13. The kernel data output unit 13 shown in the figure comprises a selector 131, a bit width conversion unit 132, and a control circuit 133.
[0068] For example, if 1024-bit wide SRAM is used as kernel memory 21 and 22, and the "number of simultaneous operations ich × number of simultaneous operations och" in mode 1 is 32, the bit width conversion unit 132 outputs the data read from the SRAM 9 times and held together so that the output is 32 bits × 9 × 32 = 9,216 bits. If the "number of simultaneous operations ich × number of simultaneous operations och" in mode 2 is 64, the bit width conversion unit 132 outputs the data read from the SRAM 2 times and held together so that the output is 32 bits × 1 × 64 = 2,048 bits. If the "number of simultaneous operations ich × number of simultaneous operations och" in mode 3 is 64, the bit width conversion unit 132 outputs the data read from the SRAM 2 times and held together so that the output is 32 bits × 1 × 64 = 2,048 bits.
[0069] In mode 3, when stride=2, the sum of kernel data per layer is 32 bits × 9 × number of ichs × number of ochs when the precision is 32 bits. For example, if the "number of simultaneous ichs" is 2 and the "number of simultaneous ochs" is 32, the first operation will be for ichs 0 to 1 of ochs 0 to 31. To perform this 3x3 convolution operation for ichs 0 to 1 of ochs 0 to 31, the bit width conversion unit 132 outputs the necessary kernel data in 9 parts.
[0070] Next, the output modes of the image data output unit 14 for each convolution process are shown below.
[0071] Mode 1: 8-bit × 60 pixels (10 × 6) × number of simultaneous operations Mode 2: 8-bit × 32 pixels (8 × 4) × number of simultaneous operations Mode 3: 8-bit × 32 pixels (extracted from 17 × 9) × number of simultaneous operations
[0072] Image memories 23 and 24 are composed of multiple SRAMs. Specifically, the bit configuration of each SRAM is 8 bits × 2 pixels (2 × 1) × number of simultaneous operations ich, and 32 of these SRAMs are installed to enable simultaneous reading of image data in modes 1 to 3. Separately, registers or SRAMs are installed to hold and output adjacent pixels in the upper left (e.g., 2 pixels × number of simultaneous operations ich), top (e.g., 32 pixels × number of simultaneous operations ich), and left (e.g., 2 pixels × number of simultaneous operations ich × 16 rows).
[0073] Next, we will explain the convolution process for each mode. Although bias data is also used in the convolution process, we will omit the explanation of bias data.
[0074] First, let's explain the convolution process in mode 1, which is 3x3 (stride=1).
[0075] In Mode 1, kernel data K(0) to K(8) of 9 × number of ich × number of ochs are used, and image data G(0,0) to G(9,5) of 60 adjacent pixels are input simultaneously. The nine kernel data K(0) to K(8) in a 3 × 3 matrix are multiplied by the image data G(x,y) at the corresponding position, and the nine multiplication results are summed up. The kernel data K(0) to K(8) are slid one pixel on the image data, and a convolution operation is performed at each position. The results of the above sum-of-products operation are given by C(1,1) to C(8,4) in the following equation.
[0076]
[0077] If there are multiple ichs, different image data and kernel data are used for each ich, and the output C of the convolution process is obtained by accumulating and summing the values of C(1,1) to C(8,4) from the above formula obtained for each ich for the number of ichs, and then adding the value of the bias data to the sum of the values obtained for each ich. out (1,1)~C out (8,4)
[0078] When the number of ochs is 1 and there are multiple ochs, different kernel data is used for each och, and the output C of the convolution process is obtained by adding the value of bias data (which differs for each och) to each of the values of C(1,1) to C(8,4) in the above formula for each och. out (1,1)~C out (8,4)
[0079] If there are multiple ichs and ochs, different image data is used for each ich. Different kernel data is used for each ich and each och. The output of the convolution process is the sum of C(1,1) to C(8,4), which are cumulatively added for each ich, plus the value of the bias data (which differs for each och).
[0080] Next, we will explain the convolution process in mode 2, which is 1x1 (stride=1).
[0081] In mode 2, kernel data K(0) of 1 × number of ichs × number of ochs is used, and the image data G(0,0) to G(7,3) of 32 adjacent pixels are input simultaneously, and the kernel data K(0) is multiplied by the image data G(x,y). The kernel data K(0) is slid one pixel on the image data, and a convolution operation is performed at each position. The result of the above multiplication is given by C(0,0) to C(7,3) in the following equation.
[0082]
[0083] If there are multiple ichs, different image data and kernel data are used for each ich, and the output C of the convolution process is obtained by accumulating and summing the values of C(0,0) to C(7,3) from the above formula obtained for each ich for the number of ichs, and then adding the value of the bias data to this value. out (0,0)~C out This results in (7,3). Note that, for example, if the number of simultaneous operations ich is 4 and the number of ich is not divisible by 4, for example, if the number of simultaneous operations ich is 4 and the number of ich is 7, then 0 can be used as the kernel data for the 8th ich.
[0084] If there are multiple ichs and ochs, different image data is used for each ich. Different kernel data is used for each ich and each och. The output of the convolution process is the sum of C(0,0) to C(7,3), which are cumulatively added for each ich, plus the value of the bias data (which differs for each och).
[0085] Next, we will explain the convolution process in mode 3, which is 3x3 (stride=2).
[0086] In mode 3, kernel data K(0) to K(8) consisting of 9 × number of ichs × number of ochs is used, and 32 pixels are extracted and used from adjacent 153 pixel image data G(0,0) to G(16,8). Multiply K(0) by the 32-pixel image data G(0,0), G(2,0), ..., G(14,6) for K(0) and store the results. Multiply K(1) by the 32-pixel image data G(1,0), G(3,0), ..., G(15,6) for K(1) and add the result to each of the results for K(0) and store the result. Multiply K(2) by the 32-pixel image data G(2,0), G(4,0), ..., G(16,6) for K(2) and add the result to each of the results up to K(1) and store the result. Multiply K(3) by the 32-pixel image data G(0,1), G(2,1), ..., G(14,7) for K(3) and add the result to each of the results up to K(2) and store the result. Similarly, multiply and add for K(4) through K(8), and add the result of the multiplication of K(8) to each of the results up to K(7).
[0087] The 32 pixels of image data for K(0) are G(0,0), G(2,0), G(4,0), G(6,0), G(8,0), G(10,0), G(12,0), G(14,0), G(0,2), G(2,2), G(4,2), G(6,2), G(8,2), G(10,2), G(12,2), G(14,2), G(0,4), G(2,4), G(4,4), G(6,4), G(8,4), G(10,4), G(12,4), G(14,4), G(0,6), G(2,6), G(4,6), G(6,6), G(8,6), G(10,6), G(12,6), G(14,6). The 32-pixel image data for K(1) consists of pixels G(1,0), G(3,0), ..., G(15,6) to the right of the 32-pixel image data for K(0), and the 32-pixel image data for K(2) consists of pixels G(2,0), G(4,0), ..., G(16,6) to the right of the 32-pixel image data for K(1). The 32-pixel image data for K(3) consists of pixels G(0,1), G(2,1), ..., G(14,7) below the 32-pixel image data for K(0). The 32-pixel image data for K(4) consists of pixels G(1,1), G(3,1), ..., G(15,7) to the right of the 32-pixel image data for K(3). The 32-pixel image data for K(5) consists of pixels G(2,1), G(4,1), ..., G(16,7) to the right of the 32-pixel image data for K(4). The 32-pixel image data for K(6) consists of pixels G(0,2), G(2,2), ..., G(14,8) below the 32-pixel image data for K(3). The 32-pixel image data for K(7) consists of pixels G(1,2), G(3,2), ..., G(15,8) to the right of the 32-pixel image data for K(6). The 32-pixel image data for K(8) consists of pixels G(2,2), G(4,2), ..., G(16,8) to the right of the 32-pixel image data for K(7).
[0088] The result of the above sum-of-products operation is C(1,1) to C(15,7) in the following equation.
[0089]
[0090] If there are multiple ichs, different image data and kernel data are used for each ich, and the output C of the convolution process is obtained by accumulating and summing the values of C(1,1) to C(15,7) from the above formula obtained for each ich for the number of ichs, and then adding the value of the bias data. out (1,1)~C out (15,7)
[0091] If the number of ochs is 1 and there are multiple ochs, different kernel data is used for each och, and the output of the convolution process is obtained by adding the value of bias data (which differs for each och) to each of the C(1,1) to C(15,7) for each och.
[0092] If there are multiple ichs and ochs, different image data is used for each ich. Different kernel data is used for each ich and each och. The output of the convolution process is the value obtained by adding the bias data (which differs for each och) to the cumulative sum of C(1,1) to C(15,7) values for each och.
[0093] Next, we will explain the configuration of the image memory and provide an example of writing pixels.
[0094] In this embodiment, the image memories 23 and 24 are configured with 32 + 2 memories, and in any of modes 1 to 3, the image data writing unit 12 writes 578 (34 × 17) adjacent pixels to the 32 + 2 memories so that the image data output unit 14 can read pixels from each memory simultaneously.
[0095] Figure 6 shows an example of writing to 32 memory locations and pixels. In Figure 6, the pixels to be written to each address in each memory location are shown. Memories 1 to 32 are physically separate memory locations. Two pixels of data are written to each address. In the pixel writing example shown in Figure 6, pixels G(1,1) to G(8,1) are written to address 0 of memory 1 to 4, pixels G(1,2) to G(8,2) are written to address 0 of memory 5 to 8, pixels G(9,2) to G(16,2) are written to address 1 of memory 1 to 4, pixels G(9,1) to G(16,1) are written to address 1 of memory 5 to 8, pixels G(17,1) to G(24,1) are written to address 2 of memory 1 to 4, pixels G(17,2) to G(24,2) are written to address 2 of memory 5 to 8, pixels G(25,2) to G(32,2) are written to address 3 of memory 1 to 4, and pixels G(25,1) to G(32,1) are written to address 3 of memory 5 to 8. In other words, pixels from the first row and the second row are written alternately in groups of 8 pixels each. Similarly, pixels G(1,3) to G(32,4) are written to addresses 0 to 3 of memory 9 to 16, pixels G(1,5) to G(32,6) are written to addresses 0 to 3 of memory 17 to 24, and pixels G(1,7) to G(32,8) are written to addresses 0 to 3 of memory 25 to 32. Pixels from row 9 onwards are written to addresses 4 and above of memory 1 to 32. Specifically, pixels from row 9 and 10 are written to addresses 4 to 7 of memory 1 to 8, pixels from row 11 and 12 are written to addresses 4 to 7 of memory 9 to 16, pixels from row 13 and 14 are written to addresses 4 to 7 of memory 17 to 24, and pixels from row 15 and 16 are written to addresses 4 to 7 of memory 25 to 32. Furthermore, to write pixels from row 17 onwards, simply increase the addresses of memory 1 to 32 and write them in the same manner.
[0096] Although not shown in the diagram, the image memory includes, in addition to the 32 memory locations mentioned above, a memory 33 for the upper edge pixels and a memory 34 for the left edge pixels for modes 1 and 3. Memory 33 stores the upper edge pixels G(-1,0), G(0,0) to G(32,0), and memory 34 stores the leftmost two columns of pixels G(-1,0), G(0,0), G(-1,1), ..., G(0,16). Registers that can read the stored pixels all at once can be used in memory 33 and memory 34.
[0097] When writing 612 (34 x 18) pixels adjacent to 32 + 2 memory locations, memory 33 stores the pixels G(-1,-1), G(0,-1) to G(32,-1), G(-1,0), G(0,0) to G(32,0) from the top two rows.
[0098] Next, we will explain how to read image data in each mode.
[0099] In mode 1, the image data output unit 14 simultaneously reads 60-pixel (10 x 6) image data from the image memories 23 and 24.
[0100] First, let's explain the reading process when 578 (34 x 17) pixels adjacent to 32 + 2 memory locations are written. The image data output unit 14 reads 60 pixel image data from G(-1,0) to G(8,5). Specifically, the image data output unit 14 reads image data from G(1,1) to G(8,1) from memory locations 1 to 4, G(1,2) to G(8,2) from memory locations 5 to 8, G(1,3) to G(8,3) from memory locations 9 to 12, G(1,4) to G(8,4) from memory locations 13 to 16, and G(1,5) to G(8,5) from memory locations 17 to 20. It also reads image data from memory location 33 for the uppermost pixels G(-1,0) to G(8,0) and from memory location 34 for the pixels in the leftmost two columns G(-1,0) to G(0,5). Furthermore, the image data of pixels G(7,0) to G(8,5) in the rightmost two columns that were read (a total of 12 pixels) is stored in a separate register or similar.
[0101] Next, the image data output unit 14 reads the 60 pixels of the image data from G(7,0) to G(16,5) on the right of the above image data. Specifically, it reads the image data from G(9,1) to G(16,1) from memories 5 to 8, G(9,2) to G(16,2) from memories 1 to 4, G(9,3) to G(16,3) from memories 13 to 16, G(9,4) to G(16,4) from memories 9 to 12, and G(9,5) to G(16,5) from memories 21 to 24. It then reads the image data of the uppermost pixels G(7,0) to G(16,0) and the pixels of the leftmost two columns G(7,0) to G(8,5) from memory 33 using a register or similar that holds the data separately. The image data of the pixels G(15,0) to G(16,5) of the rightmost two columns (a total of 12 pixels) that were read is stored in a separate register or similar. Alternatively, the registers holding the image data from G(7,0) to G(8,5) can be overwritten. The same process is performed for reading the 60-pixel image data from G(15,0) to G(24,5) and the 60-pixel image data from G(23,0) to G(32,5). After that, the value in memory 33 is rewritten to the value from G(-1,4) to G(32,4).
[0102] Next, the image data output unit 14 reads the 60 pixels of G(-1,4) to G(8,9) below the above image data. Specifically, the image data output unit 14 reads the image data of G(1,5) to G(8,5) from memory 17 to 20, G(1,6) to G(8,6) from memory 21 to 24, G(1,7) to G(8,7) from memory 25 to 28, G(1,8) to G(8,8) from memory 29 to 32, and G(1,9) to G(8,9) from memory 1 to 4. It then reads the image data of the uppermost pixels G(-1,4) to G(8,4) from memory 33 and the leftmost pixels G(-1,4) to G(0,9) from memory 34. The read pixels G(7,4) to G(8,9) from the rightmost two columns are stored in a separate register or the like. Alternatively, instead of reading the uppermost pixels G(-1,4) to G(8,4) from memory 33, G(-1,4) and G(0,4) may be read from memory 34, and G(1,4) to G(8,4) may be read from memories 1 to 32. The reading of the 60-pixel image data from G(7,4) to G(16,9), the reading of the 60-pixel image data from G(15,4) to G(24,9), and the reading of the 60-pixel image data from G(23,4) to G(32,9) may be done in the same manner.
[0103] Next, the same process is carried out for reading the 60-pixel image data from G(-1,8) to G(8,13), from G(7,8) to G(16,13), from G(15,8) to G(24,13), and from G(23,8) to G(32,13).
[0104] Next, we will explain the reading process when 612 (34 x 18) pixels adjacent to 32 + 2 memory locations are written. The image data output unit 14 reads 60 pixel image data from G(-1,-1) to G(8,4). Specifically, the image data output unit 14 reads image data from G(1,1) to G(8,1) from memories 1 to 4, G(1,2) to G(8,2) from memories 5 to 8, G(1,3) to G(8,3) from memories 9 to 12, and G(1,4) to G(8,4) from memories 13 to 16. It also reads image data from memory 33 for the top two rows of pixels G(-1,-1) to G(8,0) and from memory 34 for the left two columns of pixels G(-1,-1) to G(0,4). Furthermore, the image data of pixels G(7,-1) to G(8,4) in the rightmost two columns that were read (a total of 12 pixels) is stored in a separate register or similar.
[0105] Next, the image data output unit 14 similarly reads the 60-pixel image data from G(7,-1) to G(16,4) to the right of the above image data, as well as the 60-pixel image data from G(15,-1) to G(24,4) and the 60-pixel image data from G(23,-1) to G(32,4).
[0106] Next, the image data output unit 14 reads the 60 pixel image data from G(-1,3) to G(8,8) below the above image data. Specifically, the image data output unit 14 reads the image data from G(1,3) to G(8,3) from memory 9 to 12, G(1,4) to G(8,4) from memory 13 to 16, G(1,5) to G(8,5) from memory 17 to 20, G(1,6) to G(8,6) from memory 21 to 24, G(1,7) to G(8,7) from memory 25 to 28, G(1,8) to G(8,8) from memory 29 to 32, and G(1,9) to G(8,9) from memory 1 to 4, and reads the image data from the leftmost pixel G(-1,3) to G(0,8) from memory 34. The pixels G(7,3) to G(8,8) of the rightmost two columns that were read are stored in a separate register or similar. The same process is followed for reading the 60 pixels of image data from G(7,3) to G(16,8), the 60 pixels of image data from G(15,3) to G(24,8), and the 60 pixels of image data from G(23,3) to G(32,8).
[0107] Next, read the 60-pixel image data from G(-1,7) to G(8,12), G(7,7) to G(16,12), G(15,7) to G(24,12), and G(23,7) to G(32,12) in the same manner. Then, read the 60-pixel image data from G(-1,11) to G(8,16), G(7,11) to G(16,16), G(15,11) to G(24,16), and G(23,11) to G(32,16) in the same manner.
[0108] When writing the top row of pixels to memory 33, that is, when writing 578 (34 x 17) pixels adjacent to 32 + 2 memory locations, the image data of 60 pixels from G(23,8) to G(32,13) becomes the last read, and the image data from G(-1,14) to G(32,16) cannot be used in the convolution operation. Even if there is memory for 578 (34 x 17) pixels, only 476 (34 x 14) pixels can be used. Even if 32 x 32 pixels are made available in memory locations 1 to 32, the image data of 60 pixels from G(23,24) to G(32,29) becomes the last read, and the image data from G(-1,30) to G(32,32) cannot be used in the convolution operation.
[0109] In contrast, when writing the top two rows of pixels to memory 33, that is, when writing 612 (34 x 18) pixels adjacent to 32 + 2 memory locations, the 60 pixels of image data from G(23,27) to G(32,16) become the last read, and the entire area of memory 1 to 32 (addresses 0 to 7) can be effectively used to perform the convolution operation. Even when 32 x 32 pixels are stored in memory 1 to 32, the entire area of memory 1 to 32 (addresses 0 to 15) can be effectively used to perform the convolution operation.
[0110] In mode 2, the image data output unit 14 simultaneously reads 32-pixel (8x4) image data from the image memories 23 and 24.
[0111] For example, when the image data output unit 14 reads 32-pixel image data from G(1,1) to G(8,4), it reads image data from G(1,1) to G(8,1) from memories 1 to 4, from G(1,2) to G(8,2) from memories 5 to 8, from G(1,3) to G(8,3) from memories 9 to 12, and from G(1,4) to G(8,4) from memories 13 to 16.
[0112] When the image data output unit 14 reads the 32 pixel image data from G(9,1) to G(16,4) on the right of the above image data, it reads the image data from G(9,1) to G(16,1) from memory 5 to 8, G(9,2) to G(16,2) from memory 1 to 4, G(9,3) to G(16,3) from memory 13 to 16, and G(9,4) to G(16,4) from memory 9 to 12.
[0113] When the image data output unit 14 reads the 32 pixel image data G(1,5) to G(8,8) below the above image data, it reads the image data G(1,5) to G(8,5) from memory 17 to 20, G(1,6) to G(8,6) from memory 21 to 24, G(1,7) to G(8,7) from memory 25 to 28, and G(1,8) to G(8,8) from memory 29 to 32.
[0114] In mode 3, the image data output unit 14 simultaneously reads 32 pixels (8 x 4) of image data from image memories 23 and 24 for each K(0) to K(8).
[0115] For example, when the image data output unit 14 reads 32 pixels of image data for K(0), it reads G(2,2), G(4,2), G(6,2), G(8,2) from memory 5 to 8, G(10,2), G(12,2), G(14,2) from memory 1 to 3, G(2,4), G(4,4), G(6,4), G(8,4) from memory 13 to 16, G(10,4), G(12,4), G(14,4) from memory 9 to 11, and memory 21 to 24 Image data for G(2,6), G(4,6), G(6,6), G(8,6) is read from memory 17-19, and image data for G(10,6), G(12,6), G(14,6) is read from memory 33, and image data for the upper pixels G(0,0), G(2,0), G(4,0), G(6,0), G(8,0), G(10,0), G(12,0), G(14,0) is read from memory 34, and image data for the leftmost pixels G(0,2), G(0,4), G(0,6) is read from memory 34.
[0116] When the image data output unit 14 reads 32 pixels of image data for K(1), it reads G(1,2), G(3,2), G(5,2), G(7,2) from memories 5 to 8, G(9,2), G(11,2), G(13,2), G(15,2) from memories 1 to 4, G(1,4), G(3,4), G(5,4), G(7,4) from memories 13 to 16, and G(9,4), G(11,4), G(13,4) from memories 9 to 12. Image data for G(15,4), G(1,6), G(3,6), G(5,6), G(7,6) is read from memory 21-24, and G(9,6), G(11,6), G(13,6), G(15,6) is read from memory 17-20, and image data for the upper edge pixels G(1,0), G(3,0), G(5,0), G(7,0), G(9,0), G(11,0), G(13,0), G(15,0) is read from memory 33.
[0117] When the image data output unit 14 reads 32 pixels of image data for K(2), it reads G(2,2), G(4,2), G(6,2), G(8,2) from memories 5 to 8, G(10,2), G(12,2), G(14,2), G(16,2) from memories 1 to 4, G(2,4), G(4,4), G(6,4), G(8,4) from memories 13 to 16, and G(10,4), G(12,4), G(14,4) from memories 9 to 12. Image data for G(16,4), G(2,6), G(4,6), G(6,6), G(8,6) is read from memory 21-24, and G(10,6), G(12,6), G(14,6), G(16,6) is read from memory 17-20, and image data for the upper edge pixels G(2,0), G(4,0), G(6,0), G(8,0), G(10,0), G(12,0), G(14,0), G(16,0) is read from memory 33.
[0118] When the image data output unit 14 reads 32 pixels of image data for K(3), it reads G(2,1), G(4,1), G(6,1), G(8,1) from memories 1 to 4, G(10,1), G(12,1), G(14,1) from memories 5 to 7, G(2,3), G(4,3), G(6,3), G(8,3) from memories 9 to 12, G(10,3), G(12,3), G(14,3) from memories 13 to 15, and G(2,5 Image data for G(4,5), G(6,5), G(8,5) is read from memory 21-23 for G(10,5), G(12,5), G(14,5), G(2,7), G(4,7), G(6,7), G(8,7) from memory 25-28 for G(2,7), G(4,7), G(6,7), G(8,7), and G(10,7), G(12,7), G(14,7) from memory 29-31 for G(10,7), G(12,7), G(14,7) and image data for the leftmost pixels G(0,1), G(0,3), G(0,5), G(0,7) is read from memory 34.
[0119] When the image data output unit 14 reads 32 pixels of image data for K(4), it reads G(1,1), G(3,1), G(5,1), G(7,1) from memories 1 to 4, G(9,1), G(11,1), G(13,1), G(15,1) from memories 5 to 8, G(1,3), G(3,3), G(5,3), G(7,3) from memories 9 to 12, and G(9,3), G(11,3), G(13,3) from memories 13 to 16. The system reads image data from memory 17-20 for G(1,5), G(3,5), G(5,5), G(7,5), from memory 21-24 for G(9,5), G(11,5), G(13,5), G(15,5), from memory 25-28 for G(1,7), G(3,7), G(5,7), G(7,7), and from memory 29-32 for G(9,7), G(11,7), G(13,7), G(15,7).
[0120] When the image data output unit 14 reads 32 pixels of image data for K(5), it reads G(2,1), G(4,1), G(6,1), G(8,1) from memories 1 to 4, G(10,1), G(12,1), G(14,1), G(16,1) from memories 5 to 8, G(2,3), G(4,3), G(6,3), G(8,3) from memories 9 to 12, and G(10,3), G(12,3), G(14,3) from memories 13 to 16. The system reads image data from memory 17-20 for G(2,5), G(4,5), G(6,5), G(8,5), from memory 21-24 for G(10,5), G(12,5), G(14,5), G(16,5), from memory 25-28 for G(2,7), G(4,7), G(6,7), G(8,7), and from memory 29-32 for G(10,7), G(12,7), G(14,7), G(16,7).
[0121] When the image data output unit 14 reads 32 pixels of image data for K(6), it reads G(2,2), G(4,2), G(6,2), G(8,2) from memory 5 to 8, G(10,2), G(12,2), G(14,2) from memory 1 to 3, G(2,4), G(4,4), G(6,4), G(8,4) from memory 13 to 16, G(10,4), G(12,4), G(14,4) from memory 9 to 11, and G(2,6 The system reads image data for G(4,6), G(6,6), G(8,6) from memory 17-19, G(10,6), G(12,6), G(14,6) from memory 29-32, G(2,8), G(4,8), G(6,8), G(8,8) from memory 25-27, and reads image data for the leftmost pixels G(0,2), G(0,4), G(0,6), G(0,8) from memory 34.
[0122] When the image data output unit 14 reads 32 pixels of image data for K(7), it reads G(1,2), G(3,2), G(5,2), G(7,2) from memory 5 to 8, G(9,2), G(11,2), G(13,2), G(15,2) from memory 1 to 4, G(1,4), G(3,4), G(5,4), G(7,4) from memory 13 to 16, and G(9,4), G(11,4), G(13,4) from memory 9 to 12. The system reads image data from memory 21-24 for G(1,6), G(3,6), G(5,6), G(7,6), from memory 17-20 for G(9,6), G(11,6), G(13,6), G(15,6), from memory 29-32 for G(1,8), G(3,8), G(5,8), G(7,8), and from memory 21-28 for G(9,8), G(11,8), G(13,8), G(15,8).
[0123] When the image data output unit 14 reads 32 pixels of image data for K(8), it reads G(2,2), G(4,2), G(6,2), G(8,2) from memories 5 to 8, G(10,2), G(12,2), G(14,2), G(16,2) from memories 1 to 4, G(2,4), G(4,4), G(6,4), G(8,4) from memories 13 to 16, and G(10,4), G(12,4), G(14,4) from memories 9 to 12. The system reads image data from memory 21-24 for G(2,6), G(4,6), G(6,6), G(8,6), from memory 17-20 for G(10,6), G(12,6), G(14,6), G(16,6), from memory 29-32 for G(2,8), G(4,8), G(6,8), G(8,8), and from memory 21-28 for G(10,8), G(12,8), G(14,8), G(16,8).
[0124] Subsequently, the read image data for G(16,0) to G(16,8) (a total of 9 pixels) is stored in a separate register, and when 32 pixels are extracted from G(16,0) to G(32,8) and convolution processing is performed, this data is also used to read 32 pixels of image data for K(0) to K(8). Then, the value in memory 33 is rewritten from G(-1,8) to the value of G(32,8), and when 32 pixels are extracted from G(0,8) to G(16,16) and convolution processing is performed, this data is also used to read 32 pixels of image data for K(0) to K(8).
[0125] In this way, the image data output unit 14 can simultaneously read image data for each pixel from physically different memories 1 to 32 in any of the modes.
[0126] When the number of simultaneous operations is 4, the configuration of memories 1 to 32 is as shown in Figure 7, with 4 channels of image data arranged in parallel at the same address in the same memory. Similarly, memories 33 and 34 also arrange 4 channels of image data in parallel.
[0127] Next, we will explain an example of writing to kernel memory.
[0128] Figure 8 shows an example of writing to kernel memories 21 and 22 in mode 1. The example in the figure is for the case where the maximum number of simultaneous operation ochs is 2. In the figure, K(a,b,c) represents kernel data, where a is the och number, b is the ich number, and c is a number from 0 to 8. The kernel data is arranged so that it can be read simultaneously for each och and sequentially for each ich. For example, kernel data can be placed in separate physical memories for each och and arranged in address order for each ich. Alternatively, instead of placing kernel data in separate physical memories for each och, for example, (if the kernel data is 32 bits) a 1024-bit wide SRAM can be used so that 32 ochs worth of kernel data can be read simultaneously from a single SRAM.
[0129] In mode 1, when the number of simultaneous operations ich is 2 and the number of simultaneous operations och is 2, the kernel data output unit 13 reads kernel data K(*,0,*) for ich0 from addresses 0 to 8 and outputs all the data together. It also reads kernel data K(*,1,*) for ich1 from addresses 9 to 17 and outputs all the data together. Assuming the kernel data is 8 bits, both outputs will be 8*9*2=144 bits. When the number of simultaneous operations och is 32, the output will be 8*9*32=2,304 bits.
[0130] Figure 9 shows an example of writing to kernel memories 21 and 22 in mode 2. The example in the figure is for cases where the maximum number of simultaneous operations ich is 4 and the maximum number of simultaneous operations och is 2. In mode 2, c is only 0.
[0131] In mode 2, when the number of simultaneous operations ich is 4 and the number of simultaneous operations och is 2, the kernel data output unit 13 reads kernel data K(*,*,0) for ich0 to 3 from addresses 0 to 3 and outputs all the data together. It also reads kernel data K(*,*,0) for ich4 to 7 from addresses 4 to 7 and outputs all the data together. Assuming the kernel data is 8 bits, 8 * 4 * 2 = 64 bits will be output in both cases. When the number of simultaneous operations och is 32, 8 * 4 * 32 = 1,024 bits will be output.
[0132] The example of writing to kernel memory 21 and 22 in mode 3 is the same as in mode 1.
[0133] In mode 3, when the number of simultaneous operations ich is 4 and the number of simultaneous operations och is 2, the kernel data output unit 13 reads kernel data K(*,*,0) for ich0 to ich0 from addresses 0, 9, 18, and 27 and outputs all the data together. It reads kernel data K(*,*,1) for ich0 to ich0 from addresses 1, 10, 19, and 28 and outputs all the data together. It reads kernel data K(*,*,2) for ich0 to ich0 from addresses 2, 11, 20, and 29 and outputs all the data together. It reads kernel data K(*,*,3) for ich0 to ich0 from addresses 3, 12, 21, and 30 and outputs all the data together. It reads kernel data K(*,*,4) for ich0 to ich0 from addresses 4, 13, 22, and 31 and outputs all the data together. It reads kernel data K(*,*,5) for ich0 to ich0 from addresses 5, 14, 23, and 32 and outputs all the data together. Read kernel data K(*,*,6) for ich0-3 from addresses 6, 15, 24, and 33, and output all data together. Read kernel data K(*,*,7) for ich0-3 from addresses 7, 16, 25, and 34, and output all data together. Read kernel data K(*,*,8) for ich0-3 from addresses 8, 17, 26, and 35, and output all data together. Read kernel data K(*,*,0) for ich4-7 from addresses 36, 45, 54, and 63, and output all data together. Read kernel data K(*,*,1) for ich4-7 from addresses 37, 46, 55, and 64, and output all data together. Read kernel data K(*,*,2) for ich4-7 from addresses 38, 47, 56, and 65, and output all data together. Read kernel data K(*,*,3) for ich4-7 from addresses 39, 48, 57, and 66, and output all data together. Read kernel data K(*,*,4) for ich4-7 from addresses 40, 49, 58, and 67, and output all data together. Read kernel data K(*,*,5) for ich4-7 from addresses 41, 50, 59, and 68, and output all data together.Read kernel data K(*,*,6) for ich4-7 from addresses 42, 51, 60, and 69, and output all the data together. Read kernel data K(*,*,7) for ich4-7 from addresses 43, 52, 61, and 70, and output all the data together. Read kernel data K(*,*,8) for ich4-7 from addresses 44, 53, 62, and 71, and output all the data together. If the kernel data is 8 bits, then in each case, 8*4*2=64 bits will be output. If the number of simultaneous operations och is 32, then 8*4*32=1,024 bits will be output.
[0134] Furthermore, the memory configuration of the third embodiment can be applied to either the first or second embodiment.
[0135] Although the handling of bias data was not described in the above descriptions of the first to third embodiments, bias data may also be written to kernel memories 21 and 22 and read in the same way as kernel data. The bias data input to the arithmetic unit 15 is the same number of bias data as the number of simultaneous operation ochs. Two memories dedicated to bias data may be provided, similar to kernel memories 21 and 22. The maximum size of bias data for one layer is 4,096 bytes when the precision is 32 bits.
[0136] As described above, the computing device of this disclosure is a computing device that performs image convolution processing using kernel data, and comprises: a plurality of kernel memories 21, 22 that hold kernel data; an image memory 23 that holds image data necessary for at least one convolution process; a kernel data writing unit 11 that reads kernel data necessary for convolution processing from an external memory and writes it to the kernel memories 21, 22; an image data writing unit 12 that reads image data necessary for at least one convolution process from an external memory and writes it to the image memory 23; a kernel data output unit 13 that reads kernel data from the kernel memories 21, 22 and outputs it; an image data output unit 14 that reads image data from the image memory 23 and outputs it; and a computing unit 15 that performs convolution processing using kernel data and image data. The kernel data writing unit 11 writes kernel data to be used for different convolution processes to one of the plurality of kernel memories 21, 22, and the kernel data output unit 12 switches the kernel memories 21, 22 from which kernel data is read according to the convolution process. As a result, without using so-called cache memory, reads from kernel memory 21, 22 and image memory 23 always have the same performance, and performance degradation due to cache misses does not occur. Furthermore, by using the first kernel memory 21 and the second kernel memory 22 alternately for each layer, the writing of kernel data for the next layer can be performed simultaneously with the convolution process. As a result, waiting time is eliminated or reduced, improving the performance of the convolution process.
[0137] Although this disclosure uses YOLOv3 as an example, the method is applicable to calculations of other neural networks besides YOLOv3. It is also applicable to convolutional processes other than neural networks.
[0138] 11...Kernel data writing unit 12...Image data writing unit 13...Kernel data output unit 131...Selector 132...Bit width conversion unit 133...Control circuit 14...Image data output unit 15...Calculation unit 21, 22...Memory for kernel 23, 24...Memory for images
Claims
1. An arithmetic unit for performing image convolution processing using kernel data, comprising: a plurality of kernel memories for holding the kernel data; an image memory for holding image data necessary for at least one convolution process; a kernel data writing unit for reading the kernel data necessary for convolution processing from an external memory and writing it to the kernel memory; an image data writing unit for reading image data necessary for at least one convolution process from an external memory and writing it to the image memory; a kernel data output unit for reading and outputting the kernel data from the kernel memory; an image data output unit for reading and outputting the image data from the image memory; and an arithmetic unit for performing convolution processing using the kernel data and the image data, wherein the kernel data writing unit writes kernel data to be used for different convolution processes to one of the plurality of kernel memories; and the kernel data output unit switches the kernel memory from which to read the kernel data according to the convolution process.
2. The arithmetic device according to claim 1, comprising a plurality of image memories, wherein the image data writing unit writes image data necessary for the next convolution process to one of the image memories, and the image data output unit switches the image memory from which to read the image data for each convolution process.
3. The arithmetic device according to claim 1 or 2, wherein the image memory is composed of a plurality of physical memories such that the image data output unit can read the image data at once, and the image data writing unit writes the image data to the image memory in an arrangement such that the image data output unit can read the image data at once in any of a plurality of types of convolution processing.
4. A calculation method for performing image convolution processing using kernel data by an arithmetic unit, wherein the arithmetic unit comprises a plurality of kernel memories for holding the kernel data and an image memory for holding image data necessary for at least one convolution process, the calculation method comprising: reading the kernel data necessary for the convolution process from an external memory and writing it to the kernel memory; reading the image data necessary for at least one convolution process from an external memory and writing it to the image memory; reading the kernel data from the kernel memory and outputting it; reading the image data from the image memory and outputting it; performing convolution processing using the kernel data and the image data; writing kernel data to be used for different convolution processes to one of the plurality of kernel memories; and switching the kernel memory from which to read the kernel data according to the convolution process.
5. The calculation method according to claim 4, wherein the calculation device comprises a plurality of image memories, writes image data necessary for the next convolution process to one of the image memories, and switches the image memory from which to read the image data for each convolution process.
6. A calculation method according to claim 4 or 5, wherein the image memory is composed of a plurality of physical memories so that the image data can be read at once, and the image data is written to the image memory in an arrangement that allows the image data to be read at once in any of the plurality of types of convolution processes.