Method and system for simulating deep learning network performance in processing in memory architecture

Abstract

Provided are a deep learning network performance simulation method and system in a PIM architecture. The deep learning network performance simulation method according to an embodiment of the present invention designs a noise model simulating noise in a PIM architecture and simulates deep learning network operation performance on the basis of the noise model. In addition, a deep learning network performance simulation method according to another embodiment of the present invention checks / evaluates whether network pruning can reduce a loss in accuracy of a deep learning network due to noise generated in the PIM architecture and contribute to robust operation.

Description

Method and System for Simulating Deep Learning Network Performance in Processing In-Memory Architecture

[0001] The present invention relates to artificial intelligence technology, and more specifically, to a method and system for simulating the impact of noise that occurs when a deep learning network is loaded in a PIM (Processing In Memory) semiconductor architecture.

[0002] As deep learning technology is utilized in various industrial fields, interest in accelerating deep learning networks in artificial intelligence semiconductors is growing. The PIM architecture, which performs Multiply and Accumulation (MAC) operations of deep learning network layers within memory, is known as a next-generation deep learning processor in that it accelerates networks by controlling data movement between memory and computation units using analog signals, unlike Neural Processing Units (NPUs) or Graphic Processing Units (GPUs).

[0003] However, the signals transmitted from the PIM are the aforementioned analog signals and are sensitive to temperature and humidity. As a result, noise is generated when analog data is converted to digital data during the MAC operation process, which causes performance degradation when deep learning networks operate on the PIM architecture.

[0004] To resolve these performance degradation issues, it is necessary to measure noise values ​​generated during operation in the actual PIM architecture for each network model. However, due to the architecture's sensitivity to temperature and humidity changes, there are limitations in measuring noise values ​​that change in real time, and there is a problem in that a significant amount of time is required for the hardware to operate to perform this measurement process.

[0005] The present invention has been devised to solve the above-mentioned problems, and the objective of the present invention is to provide a method and system for simulating the operation performance of a deep learning network by designing a noise model that mimics the noise of a PIM architecture as a means to analyze the impact of noise on a deep learning network without directly measuring the noise component values ​​generated in a PIM architecture.

[0006] In addition, the objective of the present invention is to verify / evaluate whether network pruning, which is expected to mitigate the noise impact of a deep learning network in a PIM architecture, can contribute to reducing the loss of accuracy of a deep learning network caused by noise generated in a PIM architecture and operating robustly, and to provide a method for exploring an optimal network pruning technique.

[0007] A deep learning network performance simulation method according to an embodiment of the present invention for achieving the above objective comprises: a step of receiving a deep learning network; a step of lightweighting the input deep learning network; a step of adding noise to the lightweighted deep learning network; a step of obtaining an inference result of the deep learning network with added noise; and a step of analyzing the performance of the deep learning network with added noise based on the inference result.

[0008] The deep learning network is a network to be loaded into and used in PIM (Processing In Memory), and the noise may be a simulation of noise generated by changes in temperature and humidity in the PIM architecture.

[0009] Noise may be noise generated during the process of converting analog data received from memory into digital data for computation in the layers of a deep learning network in a PIM architecture.

[0010] The addition step may be to add noise to the computational data of the layers constituting the lightweight deep learning network.

[0011] The addition step may involve adding noise to the computation data through element-wise multiplication and then passing it through an activation function.

[0012] The lightweighting step may involve pruning the deep learning network.

[0013] In deep learning networks, as the number of weights that become zero through pruning increases, the influence of added noise may decrease.

[0014] The lightweighting step may involve pruning the deep learning network by assigning zeros to weights starting from the smallest weight data size for each layer, and the analysis step may involve searching for the optimal pruned deep learning network based on the analyzed performance.

[0015] The lightweighting step may involve pruning the deep learning network by removing filters of low importance layer by layer, and the analysis step may involve searching for the optimal pruned deep learning network based on the analyzed performance.

[0016] According to another aspect of the present invention, a deep learning network performance simulation system is provided, characterized by comprising: a processor that receives a deep learning network as input, lightens it, adds noise to the lightened deep learning network, obtains an inference result of the noise-added deep learning network, and analyzes the performance of the noise-added deep learning network based on the inference result; and a storage unit that provides storage space required for the processor.

[0017] According to another aspect of the present invention, a method for simulating the performance of a deep learning network is provided, characterized by comprising: a step of adding noise generated in a Processing In Memory (PIM) to a deep learning network to be loaded into the PIM; a step of obtaining an inference result of the deep learning network with the added noise; and a step of analyzing the performance of the deep learning network with the added noise based on the inference result.

[0018] According to another aspect of the present invention, a deep learning network performance simulation system is provided, characterized by comprising: a processor that adds noise generated in a deep learning network to be loaded in a Processing In Memory (PIM), obtains inference results of the deep learning network with added noise, and analyzes the performance of the deep learning network with added noise based on the inference results; and a storage unit that provides storage space required for the processor.

[0019] As described above, according to the embodiments of the present invention, by designing a noise model that simulates the noise of a PIM architecture and simulating the operation performance of a deep learning network based thereon, it is possible to analyze the impact of noise on a deep learning network without directly measuring the noise component values ​​generated in the PIM architecture.

[0020] In addition, according to embodiments of the present invention, it is possible to verify / evaluate whether network pruning can contribute to reducing the loss of accuracy of a deep learning network caused by noise in a PIM architecture and to operate robustly, and to search for optimal network pruning.

[0021] FIG. 1 is a flowchart of a deep learning network performance simulation method according to one embodiment of the present invention,

[0022] FIG. 2 is a flowchart of a deep learning network performance simulation method according to another embodiment of the present invention,

[0023] Figure 3 shows the process of calculating the importance between layers with dependencies within an MBConv Block of a MobileNetV2 network.

[0024] FIG. 4 shows the simulation results according to an embodiment of the present invention, and,

[0025] FIG. 5 is a configuration diagram of a network pruning system according to another embodiment of the present invention.

[0026] The present invention will be described in more detail below with reference to the drawings.

[0027] In an embodiment of the present invention, a method and system for simulating deep learning network performance in a PIM architecture are presented. This is a technique for simulating the operational performance of a deep learning network affected by noise by designing a noise model that simulates noise generated by changes in temperature and humidity in a PIM architecture and adding the noise to the deep learning network.

[0028] Additionally, in an embodiment of the present invention, it is verified / evaluated whether network pruning can contribute to reducing the loss of accuracy of a deep learning network caused by noise in a PIM architecture and to operating robustly, and a method for searching for optimal network pruning is also presented.

[0029] FIG. 1 is a diagram illustrating the flow of a deep learning network performance simulation method according to an embodiment of the present invention. In an embodiment of the present invention, the influence of noise generated in a PIM architecture is simulated for a deep learning network.

[0030] First, a deep learning network to be used in the PIM is input as the deep learning network to be simulated (S110). The input deep learning network consists of pre-trained weight data.

[0031] Then, a noise model that generates noise is created (S120). The noise generated by the noise model is caused by changes in temperature and humidity in the PIM and simulates the noise that occurs during the process of converting analog data received from memory into digital data for layer operations. The noise model can be implemented as a Gaussian model G(μ,σ), where μ and σ are the mean and standard deviation of the Gaussian model.

[0032] In the next step S110, test data is input into the target deep learning network, and noise generated by the noise model created in step S120 is added (S130). The noise is added to the computational data of each layer constituting the deep learning network. The result of the inference operation with noise inserted in each layer can be expressed by the following formula.

[0033]

[0034] Here, f represents the operation (Convolution, Linear MAC) function of the target layer, w represents the weight kernel data of the target layer, x represents the input activation data, G(μ,σ) represents the noise, and A represents the activation function (e.g., Relu, Silu, etc.). When the layer operation (f) is performed between the weight kernel data (w) and the input activation data (x), the noise [G(μ,σ)] generated by the noise model is multiplied element-wise by the operation output data [f(w,x)] ( : element-wise Multiplication) operation is added, and passed through activation function (A) to finally output data of the noise-inserted layer ( Creates ).

[0035] Next, the inference result of the deep learning network is obtained (S140), and the performance of the deep learning network with added noise is analyzed by comparing the inference result with the correct answer of the test data (S150). From the performance analysis result in step S150, the impact of noise on the deep learning network in the PIM architecture can be analyzed.

[0036] FIG. 2 is a diagram illustrating the flow of a deep learning network performance simulation method according to another embodiment of the present invention. In an embodiment of the present invention, the effect of noise generated in the PIM architecture is simulated for a deep learning network that has been lightweighted by pruning.

[0037] Steps S110 and S120 are identical to Steps S110 and S120 of Fig. 1 described above, so a detailed description of them is omitted.

[0038] After performing steps S110 and S120, the deep learning network input in step S110 is pruned to lighten and fine-tuned (S200). Pruning the deep learning network is based on the premise / assumption that as the number of weights that become zero through pruning increases, the influence of added noise on the deep learning network will decrease. The basis for the above premise / assumption is that, given that element-wise multiplication operations are applied to the layers of the deep learning network, if the number of weights that become zero increases, the noise that is multiplied by zero and removed increases, thereby reducing the influence of noise.

[0039] Subsequently, test data is input into the deep learning network pruned in step S200, and noise is added using the noise model generated in step S120 (S130). Inference results are obtained through the pruned deep learning network (S140), and the performance of the pruned deep learning network is analyzed by comparing the inference results with the correct answers of the test data (S150). From the performance analysis results in step S150, it can be confirmed / evaluated whether network pruning can contribute to reducing the loss of accuracy of the deep learning network caused by noise generated in the PIM architecture and to operating robustly.

[0040] Meanwhile, steps S130 to S150 differ only in whether the target deep learning network is pruned, and the method of performing each step is equivalent to steps S130 to S150 of FIG. 1, so a detailed explanation of these steps is omitted.

[0041] The pruning techniques of the S200 stage are explained in detail below. Depending on the type and specifications of the target network, unstructured and structured pruning techniques can be selectively applied.

[0042] The unstructured pruning technique assigns zero values ​​at a certain proportion to weight data, starting with the smallest data points for each layer. As the proportion of zero-assigned weights increases, the proportion of zero values ​​in the result data of layer operations (convolution / linear MAC) with the input data also increases. This contributes to mitigating the influence of noise when performing element-wise multiplication operations with noise. The pruning process gradually increases the proportion of zero values ​​in the weight kernel data for each layer during training, reducing accuracy loss and generating an unstructured pruned network model that is robust against noise.

[0043] Structural pruning techniques contribute to mitigating the influence of noise by reducing the size of the weight kernel through the removal of the entire weight data of filters with low importance values ​​at a certain ratio, based on the importance values ​​(e.g., L1-norm, L2-norm, etc.) of each filter in the layer-by-layer weight kernel data.

[0044] When calculating the importance of each layer, layers that are interconnected and have dependencies (e.g., input layers for shortcut and concatenation operations) are set as the same group, and the sum of the importance calculated for each layer is used to calculate the importance of the group and to remove filters with low importance. Specifically, Figure 3 illustrates the process of calculating the importance between dependent layers within the MBConv Block of the MobileNetV2 network. As shown, the first PW Conv (Point-Wise Convolution), DW Conv (Depth-Wise Convolution), and second PW Conv layers within the Block belong to the same group, and a value filter selected based on the calculated representative group importance is applied equally to all layers within the group to remove them.

[0045] Similar to unstructured pruning methods, the filter removal rate for each layer is gradually increased during the training process, ultimately creating a structured pruning network model that is robust to noise while reducing accuracy loss.

[0046] In this way, by repeating the subsequent steps while changing the pruning method / detail settings in step S200, it becomes possible to search for the optimal pruned deep learning network based on the analyzed performance and determine the optimal network pruning for a given target deep learning network.

[0047] Figure 4 is a diagram showing the results of simulating network performance by increasing the standard deviation of Gaussian noise simulating ADC noise of the PIM architecture in a VGG9 network, according to the simulation method according to an embodiment of the present invention.

[0048] The experimental networks are a VGG9 network model quantized (4, 3, 2-bit) and a VGG9 network quantized (4, 3, 2-bit) after applying an unstructured pruning technique. The results of accuracy performance simulations performed by increasing the noise standard deviation for each network can be analyzed as follows.

[0049] First, for the model without pruning (linear graph), it can be observed that network accuracy deteriorates as the added noise deviation increases due to the influence of large-scale noise. On the other hand, for the model with pruning applied (dotted line graph), although accuracy degradation occurred as the number of quantization bits decreased, it can be seen that the loss in accuracy was significantly reduced even as the added noise deviation increased.

[0050] Through this, it is possible to confirm that the value-keeping technique applied to the network model significantly contributed to mitigating the noise impact of the PIM architecture, and to verify the operational validity of the PIM architecture simulator.

[0051] FIG. 5 is a diagram illustrating the configuration of a deep learning network performance simulation system according to another embodiment of the present invention. The deep learning network performance simulation system according to an embodiment of the present invention can be implemented as a computing system comprising a communication unit (310), an output unit (320), a processor (330), an input unit (340), and a storage unit (350) as illustrated.

[0052] The communication unit (310) is a communication interface for connecting to an external network or external device, the output unit (320) is an output means for displaying the result of an operation performed by the processor (330), and the input unit (340) is a user interface that receives user commands and transmits them to the processor (330).

[0053] The processor (330) simulates the operational performance of the deep learning network affected by noise by designing a noise model that mimics the noise generated in the PIM architecture according to the procedure illustrated in FIG. 1 described above and adding the noise to the deep learning network, and verifies / evaluates whether network pruning can contribute to reducing the loss of accuracy of the deep learning network caused by the noise generated in the PIM architecture and operating robustly according to the procedure illustrated in FIG. 2 described above, and searches for the optimal network pruning. The storage unit (350) provides storage space necessary for the processor (330) to function and operate.

[0054] So far, preferred embodiments of a deep learning network performance simulation method and system in a PIM architecture have been described in detail.

[0055] In the above embodiment, a method for simulating the operational performance of a deep learning network due to noise influence is presented by designing a noise model that simulates noise generated by temperature and humidity changes in a PIM architecture and adding the noise to the deep learning network.

[0056] In addition, the above embodiment verified and evaluated whether network pruning could contribute to reducing the loss of accuracy of a deep learning network caused by noise in a PIM architecture and to operating robustly, and also presented a method to search for optimal network pruning.

[0057] Meanwhile, it goes without saying that the technical concept of the present invention may also be applied to a computer-readable recording medium containing a computer program that enables the device and method according to the present embodiment to perform their functions. Furthermore, the technical concept according to various embodiments of the present invention may be implemented in the form of computer-readable code recorded on a computer-readable recording medium. A computer-readable recording medium may be any data storage device that can be read by a computer and store data. For example, a computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, etc. Additionally, computer-readable code or a program stored on a computer-readable recording medium may be transmitted through a network connected between computers.

[0058] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

Claims

1. Step of receiving input to the deep learning network; A step to lighten the input deep learning network; A step of adding noise to a lightweight deep learning network; A step of obtaining the inference result of a noise-added deep learning network; A deep learning network performance simulation method characterized by including the step of analyzing the performance of a noise-added deep learning network based on inference results.

2. In Claim 1, Deep learning networks are, It is a network to be loaded into and used in PIM (Processing In Memory), and Noise is, A deep learning network performance simulation method characterized by simulating noise caused by changes in temperature and humidity in a PIM architecture.

3. In Claim 2, Noise is, A deep learning network performance simulation method characterized by noise generated during the process of converting analog data received from memory into digital data for computation in a layer of a deep learning network in a PIM architecture.

4. In Claim 1, Additional steps are, A deep learning network performance simulation method characterized by adding noise to the computational data of the layers constituting a lightweight deep learning network.

5. In Claim 4, Additional steps are, A deep learning network performance simulation method characterized by adding noise to the computational data using an element-wise multiplication operation and then passing it through an activation function.

6. In Claim 1, The lightweighting stage is, A deep learning network performance simulation method characterized by performing pruning on a deep learning network.

7. In Claim 6, Deep learning networks are, A deep learning network performance simulation method characterized by the fact that as the weights that become zero by pruning increase, the influence of added noise decreases.

8. In Claim 6, The lightweighting stage is, Pruning is performed on the deep learning network by assigning 0 to weights starting from the smallest weight data size for each layer, and The analysis stage is, A deep learning network performance simulation method characterized by exploring an optimal pruned deep learning network based on the performance being analyzed.

9. In Claim 6, The lightweighting stage is, Pruning is performed on the deep learning network by removing low-importance filters layer by layer, and The analysis stage is, A deep learning network performance simulation method characterized by exploring an optimal pruned deep learning network based on the performance being analyzed.

10. A processor that receives a deep learning network as input, performs a lightweight operation, adds noise to the lightweight deep learning network, obtains an inference result of the noise-added deep learning network, and analyzes the performance of the noise-added deep learning network based on the inference result; and A deep learning network performance simulation system characterized by including a storage unit that provides storage space required for a processor.

11. A step of adding noise generated in PIM to a deep learning network to be loaded into and used in PIM (Processing In Memory); A step of obtaining the inference result of a noise-added deep learning network; A deep learning network performance simulation method characterized by including the step of analyzing the performance of a noise-added deep learning network based on inference results.

12. A processor that adds noise generated in PIM to a deep learning network to be loaded into and used in PIM (Processing In Memory), obtains inference results of the noise-added deep learning network, and analyzes the performance of the noise-added deep learning network based on the inference results; and A deep learning network performance simulation system characterized by including a storage unit that provides storage space required for a processor.

Images