Neural Network Delta Weight Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As the number of tasks increases, the number of task-specific deep neural network models and their parameters grows linearly, making it inefficient to store and load multiple task-specific models effectively in computer vision applications.
Innovation Solution
The method involves fine-tuning a pre-trained neural network for a specific purpose, determining delta weights between the pre-trained and fine-tuned networks, compressing these delta weights through pruning and quantization, and retraining the network based on the compressed weights, with the updated delta weights being encoded and stored.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple task-specific deep neural network models are trained and stored, then the ability to perform multiple computer vision tasks is improved, but the storage space and loading time required increase linearly
Solution Approach 1:
The patent merges multiple task-specific models into a single unified model structure. Instead of storing separate models for each task, the system uses one base model that can be dynamically adapted to different tasks through selective activation of shared parameters and task-specific parameter adjustments, thereby reducing storage requirements while maintaining multi-task capability
Solution Approach 2:
The patent creates a universal base model that serves multiple functions across different tasks. The shared parameters in the base model provide universal functionality that can be adapted to various computer vision tasks, eliminating the need for separate dedicated models for each task and thus reducing overall storage space
2Adaptability or versatility
If multiple task-specific deep neural network models are trained and stored, then the ability to perform multiple computer vision tasks is improved, but the loading time increases linearly
Solution Approach 1:
By combining multiple task-specific models into a single unified model structure with shared parameters, the system loads only one model into memory regardless of the number of tasks. This dramatically reduces loading time compared to loading multiple separate models, while still enabling the system to perform multiple tasks through parameter sharing and selective activation
Solution Approach 2:
The base model is pre-trained with shared parameters that provide a foundation for multiple tasks. This preliminary training action allows the model to be quickly adapted to different tasks without requiring complete retraining or loading of separate models, thus reducing the time needed to prepare for different task executions
3Quantity of substance
If delta weights are compressed using pruning and quantization, then the data size is reduced, but the model accuracy may deteriorate
Solution Approach 1:
The patent applies parameter changes through pruning (setting small weights to zero) and quantization (reducing precision of weight representations). These parameter transformations reduce the data size of delta weights while the iterative retraining process compensates for any accuracy loss by adjusting the pruned and quantized parameters to maintain model performance
Solution Approach 2:
The patent implements feedback through iterative retraining where the model performance is continuously evaluated after compression. If accuracy deteriorates, the compression process is adjusted and retraining is performed again. This feedback loop ensures that the compressed model maintains acceptable accuracy levels while achieving significant data size reduction
Data Source
AI summary
A method with neural network compression includes: generating a second neural network by fine-tuning a first neural network, which is pre-trained based on training data, for a predetermined purpose; determining delta weights corresponding to differences between weights of the first neural network and weights of the second neural network; compressing the delta weights; retraining the second neural network updated based on the compressed delta weights and the weights of the first neural network; and encoding and storing the delta weights updated by the retraining of the second neural network.