Deep Learning Model Initialization via Shallow Historical Parameters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning models are typically trained in a batch learning manner, which is not practical for applications where data arrives in a streaming form, and requires significant storage space for pre-prepared training datasets, making it unsuitable for scenarios with limited storage and dynamic data characteristics.
Innovation Solution
Training a deep learning model using current streaming sample data with initialization parameters from a shallow learning model trained on historical data, allowing for online learning and efficient model updates as data changes over time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If batch learning is used to train deep learning models, then model training can be performed with complete datasets, but it requires significant storage space and is not suitable for streaming data
Solution Approach 1:
The patent transitions from static batch learning to dynamic online learning, where the model continuously adapts to streaming data in real-time. The learning system becomes dynamic by processing data as it arrives rather than requiring complete datasets stored in advance, thereby reducing storage requirements while maintaining training effectiveness.
Solution Approach 2:
The patent segments the training process into incremental updates based on streaming data batches. Instead of training on complete datasets at once, the model performs continuous incremental learning on segmented portions of data as they stream in, eliminating the need to store complete training datasets.
2Adaptability or versatility
If deep learning models are trained from scratch on streaming data, then online learning can be achieved, but convergence is slow and training efficiency is low
Solution Approach 1:
The patent applies preliminary action by pre-training a shallow learning model on historical data before deploying it for online learning. This preliminary training provides a good initialization for the deep learning model, so when online learning begins, the model starts from a knowledgeable state rather than random initialization, significantly accelerating convergence.
Solution Approach 2:
The patent implements nesting by embedding the shallow learning model within the deep learning model architecture. The shallow model serves as a foundational layer that provides pre-learned features, and the deep learning model builds upon this foundation. This nested structure allows the deep model to leverage the shallow model's preliminary learning, reducing the time needed for online convergence.
3Productivity
If shallow learning models are used for initial training, then convergence speed improves, but model performance may be limited
Solution Approach 1:
The patent merges the advantages of shallow learning models (fast convergence) with deep learning models (high performance) by using the shallow model as an initialization tool for the deep model. The shallow model's quick training provides a good starting point, while the deep model's greater capacity ensures high final performance, thus combining both benefits.
Solution Approach 2:
The shallow learning model performs preliminary action by providing initial weight initialization for the deep learning model. This preliminary training phase captures basic patterns quickly, and then the deep model builds upon this foundation to achieve higher performance, ensuring both fast convergence and high reliability.
Data Source
AI summary
Current streaming sample data is received. A current deep learning model is trained based on the current streaming sample data, the training including: obtaining a shallow learning model through training based on historical sample data associated with the current streaming sample data; and initializing parameters of the current deep learning model as the parameters of the shallowing learning model.