Neural Architecture Search With Single-Stage Shared-Weight Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network architecture search techniques require multiple stages of training and post-processing to achieve acceptable performance on devices with limited computational resources, leading to suboptimal accuracy and inefficiency.
Innovation Solution
A single-stage model training method that simultaneously trains a plurality of neural networks with shared parameters, allowing direct selection of a neural network that meets resource constraints without additional training, using a system to determine an architecture optimized for specific hardware devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing neural network architecture search techniques are used, then a neural network can be deployed on devices with limited computational resources, but multiple stages of training and post-processing are required leading to suboptimal accuracy and inefficiency
Solution Approach 1:
The patent segments the training process into a single-stage procedure where multiple neural network architectures are trained simultaneously with shared parameters. Each architecture is a distinct segment of the search space, but they share common weight initializations and training dynamics, eliminating the need for separate training stages and post-processing while achieving state-of-the-art accuracy.
Solution Approach 2:
The trained neural network system is universal in that the same training procedure produces multiple deployable architectures that can serve different resource constraints. A single training run generates a portfolio of models optimized for various device capabilities, eliminating the need for separate training pipelines and post-processing steps for each architecture.
2Use of energy by moving object
If a large neural network is trained to determine shared weights and then candidate architectures are selected, then resource constraints can be satisfied, but the absolute accuracies are much lower than stand-alone models requiring significant post-processing
Solution Approach 1:
The patent applies preliminary action by initializing all candidate architectures with shared weight parameters before training begins. This preliminary setup ensures that each architecture starts from an optimized parameter space, allowing them to achieve high accuracy from the first training stage without requiring subsequent post-processing or fine-tuning.
Solution Approach 2:
The system changes parameters by allowing different architectural configurations (layer depths, widths, connectivity patterns) while maintaining shared weight parameters across all candidates during training. This parameter differentiation enables each architecture to be optimized for its specific resource constraints while achieving competitive accuracy without additional training stages.
3Adaptability or versatility
If conventional techniques are used to search for architectures satisfying resource constraints, then candidate models can be identified, but computationally intensive and time consuming post-processing is required
Solution Approach 1:
The training procedure is self-service in that it automatically produces ready-to-deploy models without requiring external post-processing intervention. The single-stage training inherently optimizes each architecture for its resource constraints, and the models can be directly deployed immediately after training completes, eliminating time-consuming post-processing steps.
Solution Approach 2:
The useful action of training continues seamlessly across all candidate architectures simultaneously in a unified procedure. Rather than training one architecture at a time followed by post-processing, the system maintains continuous training across the entire architecture search space, producing deployable models as a continuous output without interruption or additional processing stages.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for selecting a neural network to perform a particular machine learning task while satisfying a set of constraints.