Block-Level Neural Architecture Search Under Hardware Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural architecture search (NAS) techniques require significant computing resources and time for computation, and existing methods like performance prediction neural networks are inefficient in optimizing neural network structures for hardware performance.
Innovation Solution
A method and apparatus that acquire performance of neural network blocks, select target blocks for improvement, train candidate blocks based on input and output data, and replace the target blocks with the trained candidate blocks, using a loss function and hardware restriction conditions to optimize neural network architecture efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural architecture search is performed using traditional methods (reinforcement learning, knowledge distillation), then a high recognition rate can be achieved, but significant computing resources and time are required
Solution Approach 1:
The patent segments the neural network into multiple blocks and performs architecture search at the block level rather than searching the entire network architecture. This division allows parallel processing and reduces the overall search space, enabling faster computation while maintaining the ability to achieve high recognition rates through selective block optimization.
Solution Approach 2:
The patent uses a performance prediction model to preliminarily evaluate candidate blocks before full training. This preliminary assessment filters out poor-performing candidates early in the process, reducing the number of expensive full training operations needed and thereby reducing total computation time while still identifying high-performing architectures.
2Reliability
If neural architecture search is performed using traditional methods, then optimal architecture can be found, but a lot of computing resources are required
Solution Approach 1:
The patent applies different optimization strategies to different blocks based on their specific characteristics and importance to the overall network performance. Critical blocks receive more thorough optimization resources while less critical blocks use lighter evaluation, allocating computing resources efficiently based on local quality requirements rather than uniform resource distribution.
Solution Approach 2:
The patent uses a performance prediction model that creates simplified copies or representations of candidate blocks for initial evaluation. These proxy models require far fewer computing resources than full training runs, allowing extensive architecture search to be performed with minimal resource consumption while still identifying promising candidates for full evaluation.
3Reliability
If the entire neural network architecture is searched, then global optimization can be achieved, but the search space becomes too large and computation becomes infeasible
Solution Approach 1:
The patent divides the global architecture search problem into multiple local block-level search problems. By searching for optimal blocks independently and then composing them into the full network architecture, the method achieves near-global optimization without the exponential complexity of searching all possible full network architectures directly.
Solution Approach 2:
The patent merges the results of independent block-level searches to construct the overall network architecture. By combining locally optimized blocks according to their functional requirements and compatibility, the method achieves effective global optimization while avoiding the computational infeasibility of exhaustive global search.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A processor-implemented neural architecture search method includes: acquiring performance of neural network blocks included in a pre-trained neural network; selecting at least one target block for performance improvement from the neural network blocks; training weights and architecture parameters of candidate blocks corresponding to the target block based on arbitrary input data and output data of the target block generated based on the input data; and updating the pre-trained neural network by replacing the target block in the pre-trained neural network with one of the candidate blocks based on the trained architecture parameters.