Method and system for structured pruning of deep learning network based on hardware characteristics

Abstract

Provided are a method and a system for structured pruning of a deep learning network based on hardware characteristics. The network pruning method according to an embodiment of the present invention generates a plurality of filter combination candidate groups by combining filters for each layer of a deep learning network to be pruned, searches for an optimal filter combination candidate group for each layer on the basis of the importance and processing time of each filter combination candidate group for each layer, and obtains a deep learning network in which each layer is configured by the discovered optimal filter combination candidate group. Accordingly, the loss of accuracy due to network pruning can be minimized to preserve the accuracy of a target network while maximizing the processing speed for each target hardware.

Description

Structural Pruning Method and System for Hardware Characteristics-Based Deep Learning Networks

[0001] The present invention relates to the lightweighting of artificial intelligence, and more specifically, to a pruning method essential for mounting a network on hardware in an intelligent cognitive system that performs recognition using a deep learning network.

[0002] As deep learning technology is utilized in various industrial fields and networks grow in size, interest in lightweighting techniques that effectively compress networks is increasing in order to overcome the difficulties of deploying networks in hardware environments. Among lightweighting techniques, pruning is a representative network compression technique that reduces the size of a deep learning network by reducing the size of its weight parameters.

[0003] Specifically, network pruning techniques can be divided into unstructured pruning, which reduces the size of a network by assigning zeros to unimportant data values ​​among the weight data of a pre-trained deep learning network, and structured pruning, which removes low-importance filters from the weight kernel data. Among these, the structured pruning method, which can reduce the number of filters in the weight data, is known to be effective for reducing the processing time of a deep learning network in a real hardware environment.

[0004] However, conventional structural pruning methods fail to reflect the actual processing time in hardware environments based on the reduction in the number of weight data filters for each layer of the network. Consequently, even with the same network model, the processing time reduction effect varies depending on the hardware, and there are limitations in identifying network structures capable of operating with optimal processing time on the target hardware through pruning.

[0005] The present invention has been devised to solve the above-mentioned problems, and the objective of the present invention is to provide a structural pruning method and system for a deep learning network that searches for an optimal filter combination based on the processing time of the deep learning network and the importance of each filter per layer in an actual hardware environment, as a measure to minimize the loss of accuracy due to network pruning and maximize the processing speed for each target hardware.

[0006] A network pruning method according to an embodiment of the present invention for achieving the above objective comprises: receiving a deep learning network to be pruned; calculating the importance of each filter for each layer of the input deep learning network; generating a plurality of filter combination candidate sets by combining filters for each layer; searching for an optimal filter combination candidate set for each layer based on the importance and processing time of each filter combination candidate set for each layer; and obtaining a deep learning network in which each layer is composed of the searched optimal filter combination candidate set.

[0007] The calculation step may be to calculate the importance of each filter based on the change in loss of the deep learning network when the filter is removed.

[0008] The generation step may involve generating multiple filter structure candidates by combining filters while reducing the filters by an integer multiple of the pruning unit.

[0009] The importance of a filter combination candidate group may be the sum of the importances calculated in the calculation step for the filters constituting the filter combination candidate group.

[0010] The processing time of a filter combination candidate may be the time expected to be spent processing input data in the corresponding layer when the deep learning network is operated on hardware when the filter combination candidate is selected.

[0011] The search step may involve referring to the processing times of filter combination candidates in a Look-up Table (LUT) containing input data processing times in the corresponding layer based on the number of unpruned filters measured through prior simulation.

[0012] The search step may involve searching for the optimal filter combination candidate with the maximum importance among the filter combination candidate groups whose processing time for each layer satisfies the target processing time.

[0013] The search phase may involve searching for a candidate set of optimal filter combinations while gradually reducing the target processing time.

[0014] The network pruning method according to the present invention may further include the step of fine-tuning the acquired deep learning network.

[0015] According to another aspect of the present invention, a network pruning system is provided, comprising: a processor that receives a deep learning network to be pruned, calculates the importance of each filter for each layer of the input deep learning network, combines filters for each layer to generate a plurality of filter combination candidate sets, searches for an optimal filter combination candidate set for each layer based on the importance and processing time of each filter combination candidate set for each layer, and obtains a deep learning network in which each layer is composed of the searched optimal filter combination candidate set; and a storage unit that provides storage space required for the processor.

[0016] According to another aspect of the present invention, a network pruning method is provided, characterized by comprising: a step of generating a plurality of filter combination candidate sets by combining filters for each layer of a deep learning network to be pruned; a step of searching for an optimal filter combination candidate set for each layer based on the importance and processing time of each filter combination candidate set for each layer; and a step of obtaining a deep learning network in which each layer is composed of the searched optimal filter combination candidate set.

[0017] According to another aspect of the present invention, a network pruning system is provided, characterized by comprising: a processor that generates a plurality of filter combination candidate sets by combining filters for each layer of a deep learning network to be pruned, searches for an optimal filter combination candidate set for each layer based on the importance and processing time of each filter combination candidate set for each layer, and obtains a deep learning network in which each layer is composed of the searched optimal filter combination candidate set; and a storage unit that provides storage space required for the processor.

[0018] As explained above, according to the embodiments of the present invention, by structurally pruning the deep learning network by searching for an optimal filter combination based on the processing time of the deep learning network and the importance of each filter per layer in an actual hardware environment, the loss of accuracy due to network pruning can be minimized, thereby preserving the accuracy of the target network and maximizing the processing speed for each target hardware.

[0019] FIG. 1 is a structural pruning method for a base deep learning network according to an embodiment of the present invention,

[0020] FIGS. 2 and FIGS. 3 are experimental results for verifying a structural pruning method according to an embodiment of the present invention,

[0021] FIG. 4 is a configuration diagram of a network pruning system according to another embodiment of the present invention.

[0022] The present invention will be described in more detail below with reference to the drawings.

[0023] In an embodiment of the present invention, a method and system for structural pruning of a deep learning network based on hardware characteristics are presented. This is a structural pruning technique for a deep learning network that can guarantee both optimal processing time and accuracy in a target hardware environment by searching for optimal filter combinations for each layer based on the processing time measured when the deep learning network operates in an actual hardware environment and the importance of each filter for each layer.

[0024] FIG. 1 is a diagram showing the flow of a structural pruning method for a base deep learning network according to one embodiment of the present invention.

[0025] As described above, first, a deep learning network consisting of pre-trained weight data is input as a deep learning network to be lightweighted by structural pruning (S110), and the importance of each filter is calculated for each layer of the deep learning network (S120).

[0026] The importance of a filter is calculated based on the change in the network's loss when the corresponding filter is removed, by measuring the network's loss depending on whether the filter is removed. Specifically, the smaller the change in the deep learning network's loss, the lower the importance of the filter; conversely, the larger the change in the deep learning network's loss, the higher the importance of the filter. L1-norm and L2-norm values ​​can be used as indicators for loss measurement.

[0027] Next, multiple filter combination candidates are generated by combining filters for each layer (S130). In step S130, filter combination candidates are generated by combining filters in all possible cases while reducing the number of filters by an integer multiple of the branching unit for each layer.

[0028] For example, if the number of filters in the layer is 32 and the branching unit is 8, the filter combination candidates are: 1) a filter combination candidate composed of 32 elements (1), 2) filter combination candidate groups that can be composed of 24 elements ( 32 C 24 1), 3) Candidate filter combinations that can be combined into 16 ( 32 C 16 1), 4) Candidate filter combinations that can be combined into 8 ( 32 It is C8.

[0029] Subsequently, based on the importance and processing time of each filter combination candidate group generated in step S130, an optimal filter combination candidate group is searched for for each layer (S140).

[0030] The importance (I) of a filter combination candidate group is the sum of the importances of the filters constituting the filter combination candidate group. This can be calculated based on the importance calculation result from step S120.

[0031] Processing time (T) refers to the time expected to be required to process input data in the corresponding layer when the filter combination candidate is selected and the deep learning network is operating in an actual hardware environment. Processing time (T) is not measured in reality, but rather refers to a processing time LUT (Look-up-Table) containing values ​​measured in advance through simulation.

[0032] The processing time LUT contains, for each layer, the results of simulating the input data processing time for that layer based on the number of unpruned filters (an integer multiple of the pruning unit). For example, in the case of a layer with 32 filters and a pruning unit of 8, the processing time for that layer is recorded as follows: with 8 filters, with 16 filters, with 24 filters, and with 32 filters. The processing times are obtained by repeatedly measuring the number of filters in the corresponding layer in the hardware environment where the deep learning network is to be run, while decreasing the number of filters by an integer multiple of the pruning unit.

[0033] The search for optimal filter combination candidates for each layer in step S140 is the process of finding the filter combination candidate with the maximum importance (I) among the filter combination candidate candidates whose processing time in the corresponding layer satisfies the target processing time. This involves finding the optimal filter combination candidate that satisfies the objective function expressed by the following linear combination equation.

[0034]

[0035] Here x i,j is the set of candidate combinations for the j-th filter of the i-th layer, N is the total number of layers in the deep learning network, M i is the number of filter combination candidates for the i-th layer, I i,j is x i,j The importance of, t i,j is x i,j The processing time of the layer composed of is a value referenced from the processing time LUT, and C is the target processing time of the deep learning work.

[0036] The optimal solution of the objective function, which is the optimal filter combination, can be obtained using the Integer Linear Programming (ILP) algorithm. Specifically, to search for a network that satisfies the target processing time (C) and minimizes the accuracy loss of the deep learning network, the pre-set target processing time (c) for each step(s) s Set ) and sequentially explore networks that satisfy this to select a pruning network that satisfies the final target processing time (C).

[0037] Step-by-step target processing time (c s ) sets the processing time T of the deep learning network to be pruned to a value that is gradually reduced by s steps (e.g., c1=0.9T, c2=0.8T, c3=0.7T...) and the target processing time (c) for each step. s When searching for filter combination candidates for the next step, an optimal network structure satisfying ) is selected, and the search is performed based on the deep learning network structure selected in the previous step. The set step(s) and the reduction in processing time for each step can be set differently depending on the specifications of the deep learning network, and when searching for filter combinations, the importance (I) is calculated using the values ​​of weight data learned as the input network for each step goes through a certain number of training epochs.

[0038] The target processing time set by repeatedly performing the step-by-step search process as described above (C, e.g., c s When =0.5T) is reached, a deep learning network composed of optimal filter combination candidates that maximize the importance of each layer while satisfying the target processing time (C) is selected as the optimal pruning network, and the search process is terminated. In this way, a deep learning network composed of optimal filter combination candidates found for each layer is obtained (S150).

[0039] The optimal pruned deep learning network obtained in the next step S150 is fine-tuned (S160). This is a process to preserve the accuracy of the target deep learning network by additionally securing accuracy through a certain number of epochs of training on the pruned deep learning network.

[0040] The pruning network generated through the structural pruning method according to an embodiment of the present invention secures high accuracy performance compared to conventional pruning methods in terms of accuracy, while simultaneously enabling operation with optimized processing time on the target hardware.

[0041] Figures 2 and 3 show experimental results for verifying a structural pruning method according to an embodiment of the present invention. The accuracy and processing time measured on the hardware (GPU) before and after performing the pruning training process on the ImageNet dataset using a commonly used MobileNetV2 network as the target network for pruning are shown. As shown in Figure 2, compared to the existing pruning method, the structural pruning method incorporating the hardware characteristics of the embodiment of the present invention can be seen to have a result that significantly reduces the processing time of the network operation while securing higher accuracy.

[0042] In addition, by analyzing the graph showing the processing time per network layer presented in Figure 3, it can be confirmed that a network structure was explored that more effectively reduces processing time by securing a lower processing time compared to the existing method (blue bar) in the front layer, where the ratio of processing time measured per layer to the total processing time of the target network before pruning (red line) is high.

[0043] FIG. 4 is a diagram illustrating the configuration of a network pruning system according to another embodiment of the present invention. The network pruning system according to an embodiment of the present invention can be implemented as a computing system comprising a communication unit (210), an output unit (220), a processor (230), an input unit (240), and a storage unit (250) as illustrated.

[0044] The communication unit (210) is a communication interface for connecting to an external network or external device, the output unit (220) is an output means for displaying the result of an operation performed by the processor (230), and the input unit (240) is a user interface that receives user commands and transmits them to the processor (230).

[0045] The processor (230) performs structural pruning on the target deep learning network by searching for layer-by-layer filter combination candidates based on processing time and filter importance according to the procedure illustrated in FIG. 1 described above. The storage unit (250) provides storage space necessary for the processor (230) to function and operate.

[0046] Up until now, a structural pruning method and system for a hardware-specific deep learning network have been described in detail with preferred embodiments.

[0047] In the above embodiment, by simulating processing time for each network layer in a hardware environment and extracting information, and by searching for filter combinations that minimize accuracy loss for each layer and maximize processing time based on this information, a method of structurally pruning a deep learning network is used to minimize accuracy loss caused by network pruning, thereby preserving the accuracy of the target network and maximizing processing speed for each hardware.

[0048] Meanwhile, it goes without saying that the technical concept of the present invention may also be applied to a computer-readable recording medium containing a computer program that enables the device and method according to the present embodiment to perform their functions. Furthermore, the technical concept according to various embodiments of the present invention may be implemented in the form of computer-readable code recorded on a computer-readable recording medium. A computer-readable recording medium may be any data storage device that can be read by a computer and store data. For example, a computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, etc. Additionally, computer-readable code or a program stored on a computer-readable recording medium may be transmitted through a network connected between computers.

[0049] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

Claims

1. Step of receiving the deep learning network to be pruned; A step of calculating the importance of each filter for each layer of the input deep learning network; A step of generating multiple filter combination candidates by combining filters for each layer; A step of searching for the optimal filter combination candidate for each layer based on the importance and processing time of each filter combination candidate for each layer; A network pruning method characterized by including the step of obtaining a deep learning network in which each layer is composed of a searched optimal filter combination candidate.

2. In Claim 1, The output stage is, A network pruning method characterized by calculating the importance of each filter based on the change in loss of a deep learning network when filters are removed.

3. In Claim 1, The generation step is, A network pruning method characterized by generating a plurality of filter structure candidate sets by combining filters while reducing the filters by an integer multiple of the pruning unit.

4. In Claim 1, The importance of filter combination candidates is, A network pruning method characterized by being the sum of the importance values ​​calculated in the calculation step for filters constituting a filter combination candidate group.

5. In Claim 4, The processing time of the filter combination candidate group is, A network pruning method characterized by the time expected to be required to process input data in the corresponding layer when the corresponding filter combination candidate group is selected during deep learning network operation in hardware.

6. In Claim 5, The exploration phase is, A network pruning method characterized by referencing the processing time of filter combination candidates in a Look-up Table (LUT) containing input data processing times in the corresponding layer based on the number of unpruned filters measured through prior simulation.

7. In Claim 5, The exploration phase is, A network pruning method characterized by searching for the optimal filter combination candidate with the maximum importance among filter combination candidate groups whose processing time satisfies the target processing time for each layer.

8. In Claim 7, The exploration phase is, A network pruning method characterized by searching for a candidate set of optimal filter combinations while gradually reducing the target processing time.

9. In Claim 1, A network pruning method characterized by further including the step of fine-tuning an acquired deep learning network.

10. A processor that receives a deep learning network to be pruned, calculates the importance of each filter for each layer of the input deep learning network, generates multiple filter combination candidate sets by combining filters for each layer, searches for an optimal filter combination candidate set for each layer based on the importance and processing time of each filter combination candidate set for each layer, and obtains a deep learning network in which each layer is composed of the searched optimal filter combination candidate set; and A network pruning system characterized by including a storage unit that provides storage space required for a processor.

11. A step of generating multiple filter combination candidate sets by combining filters for each layer of the deep learning network to be pruned; A step of searching for the optimal filter combination candidate for each layer based on the importance and processing time of each filter combination candidate for each layer; A network pruning method characterized by including the step of obtaining a deep learning network in which each layer is composed of a searched optimal filter combination candidate.

12. A processor that generates multiple filter combination candidate sets by combining filters for each layer of a deep learning network to be pruned, searches for an optimal filter combination candidate set for each layer based on the importance and processing time of each filter combination candidate set for each layer, and obtains a deep learning network in which each layer is composed of the searched optimal filter combination candidate set; and A network pruning system characterized by including a storage unit that provides storage space required for a processor.

Images