Pre-trained Neural Network Parameter Initialization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training deep neural networks (DNNs) is time-consuming, often requiring hours or days, making it inefficient for fine-tuning or adapting to larger datasets.
Innovation Solution
The method involves training a predecessor network without bias and then using the learned parameters to initialize a successor network, which trains faster while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If iterative procedures like stochastic gradient descent are used to train deep neural networks, then the network can be trained to achieve good performance, but the training time becomes very long (hours to days or weeks)
Solution Approach 1:
The patent applies preliminary action by pre-training a source network on a source dataset before using its learned parameters to initialize the target network. This preliminary training step creates a starting point that is already close to optimal, significantly reducing the iterative training time required for the target network while maintaining final accuracy.
Solution Approach 2:
The patent uses copying by transferring the learned parameters (weights) from the source network to the target network. Instead of training the target network from random initialization, the system copies the pre-trained parameters, allowing the target network to start with knowledge already embedded, thus reducing training time while preserving accuracy.
2Adaptability or versatility
If deep neural networks are trained from scratch with random initialization, then the network can learn task-specific features, but the training process becomes inflexible and time-consuming
Solution Approach 1:
The system performs preliminary action by pre-training a source network on a source dataset before using its learned parameters to initialize the target network. This preliminary training step creates a starting point that is already close to optimal, significantly reducing the iterative training time required for the target network while maintaining final accuracy.
Solution Approach 2:
The patent applies universality by training a single source network that can serve multiple target tasks. The source network's learned parameters can be transferred to different target networks for different tasks or datasets, making the training process more versatile and efficient across multiple applications rather than training each network independently from scratch.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and computer programs are presented for training a deep neural network (DNN). One method includes an operation for training a predecessor network defined for image recognition of items, where parameters of a predecessor classifier are initialized with random numbers sampled from a predetermined distribution, and the predecessor classifier utilizes an image-classification probability function without bias. The method further includes an operation for training a successor network defined for image recognition of items in a plurality of classes, where parameters of a successor classifier are initialized with parameters learned from the predecessor network, and the successor classifier utilizes the image-classification probability function without bias. Further, the method includes operations for receiving an image for recognition, and recognizing the image utilizing the successor classifier.