Deep Learning Module Incubation for Independent Training Compatibility
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training large deep learning models faces challenges on infrastructure and optimization due to high computational resource requirements and the contradiction between independency and compatibility of modularized training, leading to inefficiencies and compatibility issues.
Innovation Solution
A method using a pre-trained lightweight meta network to incubate modules, allowing independent training without cross-module communication, while ensuring compatibility through feature level alignment and input distribution matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If modularized training is used to train large models, then computational resource requirements are reduced and training independence is improved, but module compatibility deteriorates
Solution Approach 1:
The patent introduces a pre-trained lightweight meta network as an intermediary that serves as a common reference framework. During independent module training, each module is trained while the meta network provides a consistent feature space and input distribution, ensuring compatibility among modules without requiring direct communication between them.
Solution Approach 2:
The meta network is pre-trained beforehand to establish a standardized feature representation and input distribution. This preliminary action creates a common foundation that all subsequently trained modules can rely on, ensuring compatibility before the actual module training begins.
2Reliability
If end-to-end training is used to ensure model performance, then optimization is improved, but computational resource consumption increases
Solution Approach 1:
The patent segments the large deep learning model into multiple independent modules that can be trained separately. Each module is trained independently using the pre-trained meta network as a reference, eliminating the need for computationally intensive end-to-end training while maintaining model performance through the standardized feature space provided by the meta network.
Data Source
AI summary
A computer-implemented method for deep learning including obtaining a meta network including of a set of incubating modules. Each of the set includes at least one basic unit of an architecture of a deep learning network. The meta network is pre-trained on a dataset. The method includes independently training, on the dataset, a set of modules with each of the set of modules corresponding to a respective one of the set of incubating modules, with one of the set of incubating modules being substituted by one of the set of modules corresponding to the one of the set of incubating modules on the dataset for training of the one of the set of modules, wherein each module of the set includes basic unit(s) of the architecture of the deep learning network; assembling the independently trained modules; and obtaining the deep learning network that is optimized on the dataset.