Simulation-Guided Neural Network Pruning for Embedded Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing iterative pruning methods for deep neural networks determine importance based on the original network, leading to inefficient pruning and difficulty in deploying such networks on resource-constrained devices.
Innovation Solution
A simulation-guided iterative pruning method that generates a second neural network by pruning a first neural network based on a threshold, calculates gradients for each weighted value, and applies these gradients to the first neural network to iteratively refine the structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If iterative pruning is performed based on importance determined from the original network, then the pruning process is simple to implement, but the pruning efficiency and accuracy deteriorate because the importance determination does not reflect the actual structure after pruning
Solution Approach 1:
The patent performs preliminary pruning to generate a reduced network structure before determining importance. This preliminary action creates a more accurate basis for importance determination, as the importance is calculated based on the actual reduced structure rather than the original network, thereby improving pruning accuracy while maintaining implementation feasibility
Solution Approach 2:
The patent implements a feedback mechanism where the reduced network structure is fed back into the importance determination process. The importance of weighted values is recalculated based on the reduced network obtained from preliminary pruning, creating a closed-loop system that continuously improves pruning accuracy by using actual structural information from previous pruning iterations
2Adaptability or versatility
If the network structure is compressed to fit resource-constrained devices, then the deployability improves, but the network performance and accuracy may deteriorate
Solution Approach 1:
The patent changes the parameter basis for importance determination from the original network to the reduced network. By calculating importance based on the reduced network structure and adjusting the pruning threshold accordingly, the method optimizes the balance between network compression ratio and accuracy retention, enabling better deployability without significant accuracy loss
Solution Approach 2:
The patent introduces dynamic adjustment of the pruning threshold based on the reduced network structure. The threshold is not fixed but adapts to the actual network configuration after each pruning iteration, allowing the system to dynamically optimize the trade-off between compression and accuracy for resource-constrained deployment scenarios
3Productivity
If multiple pruning iterations are performed to improve compression effectiveness, then the compression ratio improves, but the computational time and complexity increase
Solution Approach 1:
The patent performs a preliminary pruning iteration to generate a reduced network structure before conducting importance determination. This preliminary action reduces the network size early in the process, thereby reducing the computational burden of subsequent importance calculations and gradient computations, which accelerates the overall pruning process while maintaining compression effectiveness
Solution Approach 2:
The patent performs pruning in iterative stages rather than attempting to achieve final compression in a single step. Each iteration performs partial pruning based on the current reduced structure, allowing the system to achieve progressive compression with manageable computational effort at each stage, ultimately reaching higher compression ratios without excessive time cost
Data Source
AI summary
The effective network compression using simulation-guided iterative pruning according to various embodiments, can be configured so that, by means of an electronic device, a first neural network is pruned on the basis of a threshold value, a second neural network is generated, a gradient for each weighted value of the second neural network is calculated, and a third neural is acquired by applying the gradient to the first neural network.