Transfer Learning Model Generation via Weight Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models require significant computational resources and training time, leading to increased energy consumption, and existing model compression techniques like pruning have limitations in scalability and task-awareness, while transfer learning struggles with high computation and parameter counts despite its efficiency in data-scarce scenarios.
Innovation Solution
Converging model compression and transfer learning techniques by reconstructing a pre-trained model, removing weights based on activation degrees, and training the model with a second dataset for a target task, using a pruning mask to minimize loss and hardware resource differences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If model compression techniques like pruning are applied to reduce model size, then storage space and computation are reduced, but task-awareness and scalability are limited
Solution Approach 1:
The method performs preliminary actions by first reconstructing the pre-trained model and then identifying weights to be removed based on activation degrees before final training. This preliminary identification and removal of insignificant weights enables task-aware model compression, resolving the contradiction between reducing parameters and maintaining task-awareness.
Solution Approach 2:
The method changes parameters by dynamically adjusting which weights are removed based on activation degrees calculated from the second dataset. This parameter change approach allows the model to adapt to specific tasks while maintaining compression, simultaneously achieving reduced parameters and improved task-awareness.
2Loss of time
If transfer learning is used to improve efficiency in data-scarce scenarios, then training time is reduced, but computation and parameter counts remain high
Solution Approach 1:
The method extracts and removes insignificant weights from the pre-trained model based on activation degrees calculated from the second dataset. This extraction process reduces the parameter count while preserving the beneficial pre-trained knowledge, thereby reducing both training time and parameter counts simultaneously.
Solution Approach 2:
Instead of removing all weights or using the complete pre-trained model, the method applies partial action by selectively removing only the insignificant weights identified through activation degree analysis. This partial removal achieves compression while maintaining the essential learned representations, reducing parameters without sacrificing transfer learning benefits.
3Measurement precision
If deep learning models are trained with high accuracy requirements, then model performance is improved, but computational resources and training time increase significantly
Solution Approach 1:
The method performs preliminary action by reconstructing the pre-trained model and identifying weights to be removed before final training on the second dataset. This preliminary processing reduces the model complexity early, enabling high accuracy training with reduced computational resources and energy consumption.
Solution Approach 2:
The method applies local quality by treating different weights differently based on their activation degrees. Weights with low activation degrees are removed while preserving weights with high activation degrees, creating a locally optimized model that maintains accuracy in critical areas while reducing overall computational requirements and energy consumption.
Data Source
AI summary
Provided are a method and system for generating a transfer learning model based on convergence of model compression and transfer learning convergence. The method of generating a transfer learning model may include reconstructing a first model that is pre-trained based on a first dataset, and generating a second model by removing at least some weights from the reconstructed first model based on a second dataset that is different from the first dataset, and generating the second model that is trained with transfer learning by using the second dataset, from the first model from which the at least some weights are removed.