Neural Network Architecture Search via Multi-Phase Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for finding optimized neural network configurations are time-consuming and computationally intensive, especially for complex applications, due to the immense number of possible configurations and the need for extensive training to evaluate their performance.
Innovation Solution
A computer-implemented method that uses approximate network morphisms to generate and evaluate neural network configuration variants, reducing the computational effort by only training the affected network parameters in a multi-phase training process, allowing for faster and more efficient architecture search.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If comprehensive training is performed to determine prediction error for complex network configurations, then evaluation accuracy is improved, but computational effort and training time increase significantly
Solution Approach 1:
The patent segments the training process into two distinct phases: a first training phase that trains only the network parameters affected by the morphism transformation, and a second training phase that performs comprehensive training. This segmentation allows the system to evaluate prediction errors with sufficient accuracy while significantly reducing the computational effort required compared to performing complete retraining for each configuration variant.
Solution Approach 2:
The first training phase serves as a preliminary action that pre-trains the affected network parameters before the comprehensive second training phase. This preliminary training provides a good initialization point that accelerates convergence in the second phase, enabling accurate prediction error evaluation with reduced total computational effort.
2Adaptability or versatility
If the number of network configuration variants is increased to cover more architectural possibilities, then solution quality is improved, but search time increases considerably
Solution Approach 1:
The method applies preliminary network morphisms to generate configuration variants before the search process begins. This preliminary generation of diverse architectural variants enables comprehensive coverage of the search space while the efficient two-phase training evaluation reduces the time required to assess each variant, thereby reducing overall search time.
Solution Approach 2:
The patent employs network morphisms that transform network configurations by changing architectural parameters such as layer dimensions, connection patterns, and neuron counts. This systematic parameter transformation allows generation of numerous configuration variants with different architectural properties, improving architectural coverage while the efficient evaluation method keeps search time manageable.
3Speed
If only affected network parameters are trained in the first phase, then training speed is improved, but training completeness may be compromised
Solution Approach 1:
The training process is segmented into two phases with distinct objectives: the first phase quickly trains only affected parameters to establish a baseline evaluation, while the second phase completes comprehensive training for full reliability. This segmentation allows the system to achieve both fast initial evaluation and complete training when needed.
Solution Approach 2:
The two-phase training process incorporates feedback mechanisms where the results from the first training phase inform the second phase. The prediction errors obtained after the first phase provide feedback that guides whether and how the second comprehensive training phase should be applied, ensuring training completeness is maintained when evaluation accuracy requires it.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
The invention relates to a method for determining a suitable network configuration for a neural network for a specified application, which is defined in the form of training data, the network configuration indicating the architecture of the neural network, comprising the following steps: a) on the basis of a current network configuration, generating (S1, S2) a plurality of network configurations to be evaluated, which differ in a part from the current network configuration, by applying approximate network morphisms; b) determining affected network parts of the network configurations; c) conducting multi-phase training (S3) of each of the network configurations to be evaluated under predefined training conditions, wherein in a first phase network parameters of a part not changed by the use of the approximate network morphism in question are disregarded in the training and in at least one further phase all network parameters are trained, d) determining (S4) a resulting prediction error (error (TNj)) for each of the network configurations to be evaluated; e) selecting (S5) the suitable network configuration in dependence on the determined prediction errors.