Dynamic Neural Network Sparsification for Edge Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network model compression techniques face challenges such as high initial training costs and performance degradation due to premature pruning, especially in resource-constrained applications like edge devices, as they often require intensive initial training for full dense models before pruning and struggle to recover rich features.
Innovation Solution
A dynamic neural network sparsification method that iteratively prunes and re-evaluates parameters, allowing for temporary pruning during initial training followed by re-activation and importance-based updates to achieve target sparsity while maintaining model performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If pruning is performed on pretrained dense models to compress the model, then model size and computation costs are reduced, but intensive initial training costs are required
Solution Approach 1:
The patent applies preliminary action by initializing the neural network with a sparse structure from the beginning rather than starting with a dense model. The sparse initialization includes setting up prune masks and performing initial pruning operations during the early training stages, so that the model learns efficient representations without requiring subsequent intensive pruning training. This preliminary sparse structure establishment avoids the need for training a full dense model first.
2Loss of time
If a sparse model is generated from initialization to avoid initial training costs, then training time is reduced, but model performance deteriorates due to limited feature learning capacity
Solution Approach 1:
The patent applies dynamics by making the sparsity structure dynamic rather than static. The prune masks are updated iteratively during training based on importance metrics, allowing the model to adapt its sparse structure as learning progresses. This dynamic adjustment enables the model to recover and learn rich features over time while maintaining sparsity benefits, resolving the performance degradation issue of static sparse initialization.
Solution Approach 2:
The patent implements feedback through iterative importance estimation and mask updates. During each training iteration, the model evaluates the importance of pruned parameters, and this feedback is used to update the prune masks for subsequent iterations. This closed-loop feedback mechanism allows the model to progressively learn which parameters are important and adjust the sparse structure accordingly, maintaining performance while benefiting from compression.
3Productivity
If parameters are pruned from the neural network model, then computation efficiency is improved, but the model loses the ability to recover rich features once pruned
Solution Approach 1:
The patent applies discarding and recovering by temporarily removing (pruning) parameters during training iterations and then recovering them in subsequent iterations based on their learned importance. The prune masks are updated iteratively, allowing parameters to be discarded when not important and recovered when they become important through learning. This dynamic discard-recover process maintains computation efficiency while preserving the ability to recover rich features.
Data Source
AI summary
Machine learning is a process that learns a neural network model from a given dataset, where the model can then be used to make a prediction about new data. In order to reduce the size, computation, and latency of a neural network model, a compression technique can be employed which includes model sparsification. To avoid the negative consequences of pruning a fully pretrained neural network model and on the other hand of training a sparse model in the first place without any recovery option, the present disclosure provides a dynamic neural network model sparsification process which allows for recovery of previously pruned parts to improve the quality of the sparse neural network model.