Neural Network Co-Training for Architecture Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural architecture search methods incur high computing costs due to the need to train each sampled architecture from scratch, even when using super-networks and sub-networks, which is inefficient and environmentally costly.
Innovation Solution
The method involves co-training a full-sized network and multiple sub-networks in parallel using supervised co-training iterations, where the full-sized network is maximized with respect to ground truth labels, and sub-networks are trained based on the output of the full-sized network, reducing the number of sampled networks and training time through upper-attentive sampling and in-place distillation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If each sampled architecture is trained from scratch, then the optimal neural network architecture can be found, but the computing cost increases significantly
Solution Approach 1:
The patent segments the training process by dividing architectures into a full-sized network and multiple sub-networks with different FLOPs. Each sub-network is trained independently with its own optimizer and loss function, allowing parallel training without requiring to train each architecture from scratch separately. This segmentation enables efficient exploration of architecture space while reducing redundant computation.
Solution Approach 2:
The patent creates a universal training framework that can handle multiple architectures simultaneously. The full-sized network and sub-networks share the same backbone structure and can be trained in parallel using the same infrastructure. This multi-functionality allows the system to optimize multiple architectures with different hardware constraints in a single training run, significantly reducing computing costs.
2Productivity
If multiple architectures are trained in parallel, then the training time is reduced, but the system complexity increases
Solution Approach 1:
The patent segments the network into a full-sized network and multiple sub-networks, each with distinct FLOPs and training objectives. This segmentation allows independent optimization of each network while maintaining a unified training framework. The modular structure makes it easier to manage complexity compared to training completely independent architectures.
Solution Approach 2:
The patent introduces a teacher-student framework where the full-sized network acts as a teacher and sub-networks act as students. This intermediary relationship provides a structured way to manage parallel training, with the teacher network guiding the training of student networks through knowledge distillation. This mediator structure simplifies the coordination of multiple parallel training processes.
3Use of energy by moving object
If sub-networks share weights with the super-network, then the training cost is reduced, but the ability to optimize for specific hardware constraints is limited
Solution Approach 1:
The patent segments the weight sharing strategy by allowing partial weight sharing between the full-sized network and sub-networks. Instead of complete weight sharing, each sub-network has its own trainable parameters while sharing the backbone structure. This enables both cost efficiency through shared initialization and adaptability through independent optimization for specific hardware constraints.
Solution Approach 2:
The patent applies local quality by allowing different sub-networks to have different levels of weight independence based on their specific hardware constraints. Sub-networks targeting different FLOPs can have different degrees of parameter sharing, enabling localized optimization for specific deployment scenarios while maintaining overall training efficiency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system and method for training a neural network. In some embodiments the method includes training a full-sized network and a plurality of sub-networks, the training including performing a plurality of iterations of supervised co-training, the performing of each iteration including co-training the full-sized network and a subset of the plurality of sub-networks.