Modular Neural Network Training Efficiency via Weight-Based Data Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In existing methods for constructing modular neural networks, data is input to all neural network modules during machine learning, leading to wasted calculation processing on modules with zero weight, and preventing the application of mini-batch processing, which is essential for improving learning efficiency.
Innovation Solution
The proposed solution involves classifying input data into groups based on the output weights of neural network modules and generating mini-batches such that data from the same group is included together, allowing for efficient mini-batch processing by limiting calculation to specific modules during the deterministic training phase.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is input to all neural network modules during machine learning, then all modules can be trained, but calculation processing is wasted on modules with zero weight
Solution Approach 1:
The patent segments the training process into two distinct phases: a probabilistic training phase where all modules receive data to establish initial weights, and a deterministic training phase where only selected modules with non-zero weights receive data. This segmentation allows the system to maintain training completeness for all modules while eliminating wasted calculation on inactive modules during the deterministic phase.
Solution Approach 2:
The patent implements dynamic module selection based on weight values. During the deterministic training phase, modules are dynamically selected for data input based on whether their weights are non-zero. This dynamic approach allows the training system to adaptively direct computational resources only to active modules, reducing overall energy consumption while maintaining effective training.
2Reliability
If data is input to all neural network modules, then all modules participate in learning, but mini-batch processing cannot be applied
Solution Approach 1:
The patent divides the training process into probabilistic and deterministic phases, enabling mini-batch processing during the deterministic phase. In this phase, data is segmented into mini-batches that are processed simultaneously through the selected modules, significantly improving learning efficiency and productivity while maintaining comprehensive module participation through the two-phase structure.
3Reliability
If all neural network modules are processed, then comprehensive training is achieved, but training time increases
Solution Approach 1:
The patent performs preliminary action during the probabilistic training phase by initializing weights for all modules and establishing which modules should be active. This preliminary setup enables the subsequent deterministic training phase to process only relevant modules with non-zero weights, reducing training time while maintaining comprehensive training through the initial probabilistic phase.
Solution Approach 2:
The patent implements dynamic training where the set of active modules changes based on weight values. During deterministic training, only modules with non-zero weights are processed, creating a dynamic training regimen that reduces time consumption on inactive modules while maintaining training comprehensiveness through the two-phase approach.
Data Source
AI summary
A non-transitory computer-readable recording medium storing an information processing program for causing a processor to execute processing including: classifying input data into one or more groups based on a weight of output of each neural network module in a case where data input in training by machine learning is performed for a plurality of neural network modules; and generating, in machine learning processing after the classification, a mini-batch of the input data such that pieces of the input data included in the same group are included in the same mini-batch.