Automated Fine-Tuning of Pre-Trained Neural Networks by Dataset Divergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for selecting pre-trained neural networks for transfer learning and configuring hyper-parameters are time-consuming and prone to errors, particularly due to naive learning rate configurations, leading to inefficient and costly brute-force assessment techniques.
Innovation Solution
An automated system calculates divergence values between source and target datasets to determine optimal learning rates for each layer of a pre-trained neural network, using symmetric Kullback-Leibler divergence to refine and repurpose the network for new tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If brute-force assessment techniques are used to select pre-trained neural networks and configure hyper-parameters, then comprehensive evaluation can be performed, but the process becomes time-consuming and computationally costly
Solution Approach 1:
The system performs self-assessment by automatically calculating divergence values between source and target datasets to determine optimal learning rates, eliminating the need for manual brute-force hyperparameter tuning while maintaining evaluation quality
Solution Approach 2:
The invention changes the approach from trial-and-error hyperparameter selection to a parameter-based method where learning rates are determined by calculating divergence values (using symmetric Kullback-Leibler divergence) between datasets, transforming the tuning process into a systematic parameter optimization approach
2Ease of manufacture
If naive learning rate configurations are used, then the process is simple to implement, but the performance and efficiency of transfer learning deteriorates
Solution Approach 1:
The system introduces divergence values as an intermediary metric that bridges source and target datasets, using symmetric Kullback-Leibler divergence to calculate optimal learning rates. This intermediary approach provides a systematic method that improves efficiency while remaining computationally straightforward
3Productivity
If automated fine-tuning using divergence values is implemented, then hyper-parameter tuning efficiency is improved, but the system complexity increases
Solution Approach 1:
The invention replaces manual trial-and-error tuning (mechanical process) with an automated computational system that calculates divergence values. This substitution eliminates repetitive manual operations while introducing a systematic automated process that, although more complex, provides significant efficiency gains
Data Source
AI summary
In an embodiment, a method for fine-tuning a pre-trained neural network for transfer learning, the method comprising obtaining a first target feature vector from a first layer of a pre-trained neural network responsive to a first target data element of a target dataset passing therethrough, obtaining a first source feature vector associated with the first layer of the pre-trained neural network, calculating a first divergence value for the first layer of the pre-trained neural network based at least in part on the first target feature vector and the first source feature vector, and setting a learning rate for the first layer of the pre-trained neural network based at least in part on the first divergence value.