Neural Network Output Layer Adjustment for Transfer Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transfer learning in neural networks faces challenges such as overfitting and decreased recognition accuracy due to the lack of a clear criterion for defining suitable ranges of lower layers and the need for extensive reconfiguration of upper layers, especially when dealing with small-scale transfer target data items.
Innovation Solution
A transfer learning apparatus that includes a transfer target data evaluator and an output layer adjuster, which assigns output units based on frequency distributions to minimize configuration changes and weight updates, using pre-trained neural network configurations and weights from source data items to adapt to target data items.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If upper layers are replaced with newly configured adaptive layers and trained by using transfer target data items, then adaptability to transfer target data items is improved, but device complexity and time consumption increase
Solution Approach 1:
The neural network is divided into lower layers (fixed configuration) and upper layers (adaptive configuration). The lower layers maintain their original configuration while the upper layers are selectively replaced and trained for transfer target data items. This segmentation allows adaptability to be achieved without redesigning the entire network architecture.
Solution Approach 2:
The lower layers are pre-trained using transfer source data items before transfer learning. This preliminary training establishes a foundation that can be reused for transfer target data items, eliminating the need to retrain the entire network and reducing configuration complexity during transfer learning.
2Adaptability or versatility
If upper layers are replaced with newly configured adaptive layers, then adaptability to transfer target data items is improved, but training time increases
Solution Approach 1:
The lower layers are pre-trained using transfer source data items before transfer learning. This preliminary training establishes a foundation that can be reused for transfer target data items, eliminating the need to retrain the entire network and reducing configuration complexity during transfer learning.
Solution Approach 2:
The neural network is divided into lower layers (fixed configuration) and upper layers (adaptive configuration). The lower layers maintain their original configuration while the upper layers are selectively replaced and trained for transfer target data items. This segmentation allows adaptability to be achieved without redesigning the entire network architecture.
3Measurement precision
If the entire neural network is retrained by using transfer target data items, then recognition accuracy for transfer target data items is improved, but loss of time and computational resources increase
Solution Approach 1:
The neural network is divided into lower layers (fixed configuration) and upper layers (adaptive configuration). The lower layers maintain their original configuration while the upper layers are selectively replaced and trained for transfer target data items. This segmentation allows adaptability to be achieved without redesigning the entire network architecture.
Solution Approach 2:
The lower layers are pre-trained using transfer source data items before transfer learning. This preliminary training establishes a foundation that can be reused for transfer target data items, eliminating the need to retrain the entire network and reducing configuration complexity during transfer learning.
4Device complexity
If lower layers are used as general-purpose feature extractor without modifying configuration, then device complexity is reduced, but adaptability to different data types decreases
Solution Approach 1:
The neural network is divided into lower layers (fixed configuration) and upper layers (adaptive configuration). The lower layers maintain their original configuration while the upper layers are selectively replaced and trained for transfer target data items. This segmentation allows adaptability to be achieved without redesigning the entire network architecture.
Solution Approach 2:
The lower layers are designed as a universal feature extractor that can process both transfer source data items and transfer target data items. By maintaining their configuration unchanged, they serve multiple purposes: extracting features from source data during pre-training and from target data during transfer learning, thus achieving multi-functionality without increasing complexity.
Data Source
AI summary
A transfer learning apparatus includes a transfer target data evaluator and an output layer adjuster. The transfer target data evaluator inputs a plurality of labeled transfer target data items each assigned a label of a corresponding evaluation item from among one or more evaluation items to a neural network apparatus having been trained by using a plurality of labeled transfer source data items and including in an output layer output units, the number of which is larger than or equal to the number of evaluation items, and obtains evaluation values output from the respective output units. The output layer adjuster preferentially assigns, to each of the one or more evaluation items, an output unit from which the evaluation value having the smallest difference from the label of the evaluation item is obtained with a higher frequency, as an output unit that outputs the evaluation value of the evaluation item.