Block-wise Neural Architecture Search Using Guided Algorithm
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural architecture search (NAS) techniques face challenges in efficiently optimizing deep neural network designs due to large and complex search spaces, requiring significant computational resources and relying heavily on user intuition and knowledge, while also failing to effectively reuse previous search results.
Innovation Solution
A sample-based and block-wise NAS approach that utilizes a divide-and-conquer method to optimize search spaces through block-level hyperparameter optimization, knowledge distillation, and multi-objective optimization, allowing for more diverse and hardware-aware search spaces, and storing intermediary data for future reuse.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional NAS techniques search through large and complex search spaces to optimize deep neural network designs, then more comprehensive architecture optimization is achieved, but significant computational resources and time are required
Solution Approach 1:
The patent divides the neural network architecture into discrete blocks or modules, each containing a limited set of pre-defined operations. This segmentation transforms the continuous, complex search space into a discrete, manageable structure where each block can be independently optimized and combined, significantly reducing the computational burden while maintaining comprehensive search capability.
Solution Approach 2:
The patent pre-defines a set of candidate blocks with various operations (convolutional, pooling, normalization, activation) and their hyperparameters before the search process. This preliminary preparation of building blocks eliminates the need to search through all possible architectures from scratch, reducing computational resources while ensuring comprehensive optimization through the guided search algorithm.
2Ease of operation
If conventional NAS techniques perform layer-by-layer search to optimize network structure, then detailed architecture control is achieved, but the search process becomes extremely computationally challenging for deep networks
Solution Approach 1:
Instead of searching layer-by-layer through the entire deep network, the patent segments the search into block-level units. Each block is a self-contained module with predefined operations, allowing the search algorithm to operate at a coarser granularity. This segmentation reduces the computational complexity of deep network search while maintaining detailed control through the composition of multiple blocks.
Solution Approach 2:
The patent introduces a new dimension to the search process by organizing architectures as compositions of blocks rather than sequential layer-by-layer structures. This dimensional change from linear layer search to block composition search reduces the search space complexity while preserving architectural control precision through the structured combination of blocks.
3Adaptability or versatility
If conventional NAS techniques rely on user intuition and knowledge to guide search, then search direction is improved, but automation and objective optimization are reduced
Solution Approach 1:
The patent implements a guided search algorithm that uses feedback from evaluating block configurations against predefined objectives (accuracy, computational cost, hardware constraints). The algorithm automatically adjusts the search based on performance metrics, eliminating the need for user intuition while maintaining objective optimization through systematic evaluation and selection of optimal block combinations.
4Reliability
If conventional NAS techniques search through all possible architectures to ensure comprehensive optimization, then optimal architecture is achieved, but previous search results cannot be effectively reused
Solution Approach 1:
By segmenting the architecture into reusable blocks with predefined operations, the patent enables efficient reuse of search results. Blocks that have been evaluated and optimized in previous searches can be directly reused in new architecture compositions without re-searching, significantly reducing time while maintaining comprehensive optimization through the systematic combination of validated blocks.
Data Source
AI summary
A computing system divides an initial seed network into a plurality of blocks to form a network search space that includes candidate neural architectures. For each block in the plurality of blocks, sample-based search spaces are defined. Each sample-based search space includes candidate block configurations. The candidate block configurations are determined by determining candidate block configurations that minimize a block-wise knowledge distillation loss. A first set of block configurations that are Pareto optimal block configurations are determined from the candidate block configurations in the sample-based search spaces. Sub-super-net search spaces for each block configuration in the first set of block configurations are determined. Using input training data, each of the sub-super-net search spaces are trained to generate a trained candidate models. An optimized neural architecture is determined by determining a first trained candidate model of the trained candidate models that minimizes a knowledge distillation loss of the trained candidate models.