Layerwise Multi-Objective Neural Search Under Compute Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural architecture searches for machine-learned models require substantial compute resources and have exponentially scaling complexity, making optimization of large models prohibitively difficult, especially for devices with limited hardware resources.
Innovation Solution
A layerwise multi-objective neural architecture search approach that constructs search spaces in a layerwise manner, selecting candidate model layers based on cost metrics and grouping them to optimize machine-learned models with polynomial complexity, reducing the required compute resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional neural architecture search is used to optimize machine-learned models, then model optimization can be achieved, but compute resources and complexity scale exponentially making it prohibitively difficult for large models
Solution Approach 1:
The patent divides the neural network into discrete layers, where each layer is independently optimized by selecting from predefined layer types (e.g., convolutional, fully connected, pooling layers). This segmentation transforms the continuous architecture search into a discrete combinatorial problem that can be solved with polynomial complexity by iterating through layer combinations systematically rather than exhaustively searching the entire search space.
2Reliability
If conventional neural architecture search is used, then optimized architectures can be found, but substantial compute resources and time are required
Solution Approach 1:
The patent pre-defines a set of valid layer types and their corresponding hyperparameter configurations before the optimization process begins. By preparing these building blocks in advance, the system avoids the need to evaluate every possible architecture configuration from scratch, instead combining pre-valid layer combinations to achieve optimized architectures with reduced compute expenditure.
3Reliability
If larger models are optimized, then better performance can be achieved, but the optimization becomes infeasible due to model size and resource requirements
Solution Approach 1:
The patent implements a dynamic optimization process that adapts the search strategy based on the specific requirements of each layer and the overall model constraints. The system can adjust the granularity of search, the set of allowed layer types, and the optimization depth dynamically, enabling efficient optimization of large models by focusing computational effort where most impactful rather than treating all layers uniformly.
Data Source
AI summary
A search option can be selected for each model layer N of a plurality of model layers M. based on the model layer N, the one or more search options can be used to construct one or more candidate model layers for a model layer N+1 of the plurality of model layers. The one or more candidate model layers are respectively associated with one or more cost metrics. A cost metric is indicative of a cost associated with inclusion of the candidate model layer in an optimized machine-learned model. An optimized machine-learned model comprising M model layers can be constructed based on a cost function. The cost function maximizes an accuracy of the optimized machine-learned model subject to a sum of the cost metrics associated with each candidate model layer included in the optimized machine-learned model being less than a maximum cost.