Factorized Hierarchical Search for Resource-Constrained Mobile CNNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural architecture search techniques face challenges in efficiently designing resource-constrained neural networks due to large and complex search spaces, which are computationally impractical and lack layer diversity, leading to suboptimal performance on mobile devices.
Innovation Solution
A factorized hierarchical search space is introduced, partitioning network layers into blocks with independent sub-search spaces for parameters like number of layers, operations, and filter sizes, enabling diversity and reducing search complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing neural architecture search techniques are used to design resource-constrained neural networks, then the search space covers all possible architectures, but the search becomes computationally impractical due to large and complex search spaces
Solution Approach 1:
The patent segments the neural network architecture into multiple independent blocks, where each block can be independently searched and configured. This segmentation divides the large complex search space into smaller manageable sub-search spaces, making the overall search computationally tractable while still covering diverse architecture possibilities.
Solution Approach 2:
The patent applies local quality by allowing different blocks to have different searchable parameters and configurations. Each block can be optimized with local variations in architecture parameters, enabling diverse and adaptive network designs without requiring exhaustive search of the entire architecture space.
2Adaptability or versatility
If existing neural architecture search techniques are used, then all possible architectures are considered, but layer diversity is lacking leading to suboptimal performance
Solution Approach 1:
By dividing the network into multiple independent blocks with independent search spaces, the patent enables each block to explore diverse architectural configurations. This segmentation allows layer diversity to emerge naturally as different blocks can be configured with different numbers of layers, types of operations, and other parameters, leading to more diverse and performance-optimized architectures.
3Productivity
If the search space is reduced to make searching faster, then computational resources are saved, but the ability to find optimal architectures is compromised
Solution Approach 1:
The segmentation of the search space into independent blocks with smaller sub-search spaces enables faster searching within each block while maintaining the ability to find optimal configurations. The independent block structure ensures that the product of individual block optimizations achieves near-optimal or optimal overall architecture without requiring exhaustive search of the entire space.
Solution Approach 2:
By allowing different blocks to have different searchable parameters and search depths, the patent enables localized optimization in regions of the search space that are most critical for performance. This local quality approach ensures that computational resources are focused on finding optimal configurations where they matter most, maintaining optimization quality while improving search efficiency.
Data Source
AI summary
The present disclosure is directed to an automated neural architecture search approach for designing new neural network architectures such as, for example, resource-constrained mobile CNN models. In particular, the present disclosure provides systems and methods to perform neural architecture search using a novel factorized hierarchical search space that permits layer diversity throughout the network, thereby striking the right balance between flexibility and search space size. The resulting neural architectures are able to be run relatively faster and using relatively fewer computing resources (e.g., less processing power, less memory usage, less power consumption, etc.), all while remaining competitive with or even exceeding the performance (e.g., accuracy) of current state-of-the-art mobile-optimized models.