Neural Network Transfer Learning via Relation Vector Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transfer learning in neural networks often faces issues such as overfitting and decreased identification accuracy, particularly when adapting to target learning data with small datasets, due to the lack of a clear standard for determining the range of lower layers and the need to construct new upper layers.
Innovation Solution
A learning apparatus that uses a source neural network with identical weighting values and structure information to adapt a target neural network, generating relation vectors from output vectors to associate them with answer labels, thereby avoiding changes to the source network and preventing overfitting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If new upper layers are constructed and learning is performed using target learning data, then the neural network can adapt to the target domain, but overfitting occurs and identification accuracy decreases
Solution Approach 1:
The patent creates a second neural network that copies the structure and weighting values of the first neural network (source domain) exactly. This copying approach allows the network to leverage pre-trained features without the instability of constructing new layers, thereby maintaining identification accuracy while adapting to the target domain through relation vector generation rather than weight modification.
Solution Approach 2:
The patent segments the neural network adaptation process into two independent parts: (1) the first neural network remains frozen with its source domain weights, and (2) a second neural network is created as a copy to handle target domain data. This segmentation prevents overfitting by isolating the adaptation process from the source network's learned weights.
2Adaptability or versatility
If the source neural network is modified to adapt to target learning data, then the network can be applied to the target domain, but the source network's learned knowledge is lost
Solution Approach 1:
Instead of modifying the source neural network directly, the patent creates a second neural network as an exact copy with identical structure and weighting values. This copying enables target domain adaptation while preserving the source network's learned knowledge intact, allowing both networks to coexist with their respective domain specializations.
Solution Approach 2:
The second neural network acts as an intermediary between the source neural network and the target domain data. It receives the same input data, processes it through copied weights, and generates outputs that are then used to create relation vectors for target domain classification, thus mediating the adaptation process without affecting the source network.
3Adaptability or versatility
If transfer learning is performed by replacing upper layers, then the network can learn target domain features, but the process becomes complex and requires determining the range of lower layers
Solution Approach 1:
The patent simplifies the transfer learning process by copying the entire first neural network structure including all layers, rather than requiring manual determination of which lower layers to retain. This eliminates the complexity of layer range selection while still enabling target domain adaptation through the relation vector generation mechanism.
Solution Approach 2:
The second neural network serves multiple functions: it acts as a feature extractor using copied source weights, a generator of relation vectors for target domain classification, and a bridge between source and target domains. This multi-functionality eliminates the need for complex layer selection while achieving target domain adaptability.
Data Source
AI summary
A learning apparatus includes at least one memory and at least one circuit. The circuit (a) obtains a first neural network that has learned by using source learning data and obtains target learning data, the target learning data including a plurality of first data items each of which is given a first label and a plurality of second data items each of which is given a second label, (b) obtains a plurality of first output vectors by inputting the plurality of first data items to a second neural network and obtains a plurality of second output vectors by inputting the plurality of second data items to the second neural network, and (c) generates a first relation vector corresponding to the first label by using the plurality of first output vectors and generates a second relation vector corresponding to the second label by using the plurality of second output vectors.