Modular Neural Network Training Without Backpropagation Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of efficiently training neural network models, particularly self-supervised learning (SSL) models, is exacerbated by the increasing scale of network parameters, leading to prolonged training times and difficulties in model training due to pattern or dimension crashes.
Innovation Solution
A neural network model training method that initializes an information processing model and a reference model, allowing each processing module to be trained separately and in parallel, using masked samples to update parameters based on output results from both models without requiring gradient backward propagation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the number of network layers and network parameters are increased to improve model capability, then the model can handle more complex information processing tasks, but the training time is prolonged and training difficulty increases
Solution Approach 1:
The patent divides the neural network into multiple independent processing modules, each capable of being trained separately. This segmentation allows parallel training of different modules, significantly reducing the time required to train the entire model while maintaining the model's overall capability to handle complex tasks.
Solution Approach 2:
The patent enables training of individual processing modules independently without requiring completion of all module training first. This partial action approach allows the system to train and deploy modules as needed, reducing total training time while maintaining sufficient model capability through selective module training.
2Reliability
If traditional forward propagation and backward propagation methods are used to train the entire neural network, then all processing modules are updated, but the training efficiency is low due to sequential processing requirements
Solution Approach 1:
The patent segments the training process into independent module-level training units. Each processing module can be trained separately using its own training data and loss function, eliminating the need for sequential forward and backward propagation across the entire network. This segmentation dramatically improves training efficiency while ensuring each module achieves reliable training completion.
Solution Approach 2:
Each processing module is equipped with its own training capabilities, including independent parameter updates and loss function evaluation. This self-service approach allows modules to train autonomously without waiting for the entire network training cycle, significantly improving overall training efficiency while maintaining training reliability through individual module validation.
3Measurement precision
If network parameters of all processing modules must be clarified before updating current module parameters, then parameter updates are accurate, but the training process becomes inefficient due to dependency requirements
Solution Approach 1:
The patent segments parameter updates into independent module-level operations. Each processing module maintains its own parameter state and update mechanism, allowing accurate parameter updates within each module without requiring knowledge of other module's parameter states. This segmentation achieves both high measurement precision for parameter updates and improved productivity by eliminating inter-module update dependencies.
Data Source
AI summary
Embodiments of the present disclosure disclose a neural network model training method and apparatus, a computer program product, and a storage medium. The method includes: initializing an information processing neural network model and a reference neural network model corresponding to the information processing neural network model, the information processing neural network model including a plurality of processing modules, and the reference neural network model including a plurality of reference modules; and updating a parameter of the information processing neural network model and a parameter of the reference neural network model through a plurality of iterations.