Evolutionary Search Space Exploration for Deep Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AutoML approaches for deep learning are limited in their ability to effectively explore search spaces for neural network architectures and hyperparameters, which affects the accuracy of model selection for specific training datasets.
Innovation Solution
The method involves generating new search spaces using an evolutionary algorithm and mutation types to alter hyperparameters, allowing for the exploration of different neural network architectures and strategies, with decision trees guiding the selection process to optimize performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If existing AutoML approaches are used to explore search spaces for neural network architectures andhyperparameters, then the selection process is simplified, but the accuracy of model selection deteriorates
Solution Approach 1:
The search space exploration is segmented into multiple hierarchical levels: (1) meta-feature extraction from datasets, (2) initial search space definition with architecture representations, (3) evolutionary algorithm-based search space generation, (4) mutation operations onhyperparameters, and (5) performance evaluation. This segmentation allows each component to be optimized independently while maintaining overall system accuracy.
Solution Approach 2:
The patent introduces a new dimension of search space exploration by generating multiple candidate search spaces through evolutionary algorithms rather than relying on a single fixed search space. This dimensional expansion allows the system to explore diverse architecture configurations and hyperparameter combinations, significantly improving model selection accuracy while maintaining ease of operation through automated search.
2Measurement precision
If the search space is expanded to include more neural network architectures andhyperparameters, then the accuracy of model selection is improved, but the complexity of the system increases
Solution Approach 1:
The search space is made dynamic through evolutionary algorithms that iteratively generate and refine candidate architectures. Instead of statically defining all possible architectures upfront, the system dynamically evolves the search space based on performance feedback, allowing comprehensive exploration without manual complexity management. The mutation operations onhyperparameters further enhance this dynamic adaptation.
Solution Approach 2:
The patent systematically varies multiple parameters including architecture representation types, hyperparameter ranges, mutation rates, and evolutionary algorithm parameters. By controlling and optimizing these parameters, the system manages search space complexity while maximizing model selection accuracy. Meta-feature-based parameter selection automatically adjusts search space characteristics based on dataset properties.
3Reliability
If meta features and evolutionary algorithms are used to generate new search spaces, then the performance of deep learning models is enhanced, but the computational resources required increase
Solution Approach 1:
Meta-features are extracted and analyzed beforehand to guide the evolutionary search process. This preliminary action allows the system to focus computational resources on promising search spaces and architectures, reducing overall computational requirements while maintaining high model performance. The initial search space definition based on meta-features prevents wasted computation on unlikely candidates.
Solution Approach 2:
The evolutionary algorithm incorporates feedback loops where model performance evaluations inform subsequent search space generation and mutation operations. This feedback mechanism efficiently directs computational resources toward high-performing architectures, avoiding unnecessary computation on poor candidates and optimizing the balance between model performance and resource consumption.
Data Source
AI summary
Aspects of the invention include systems and methods to obtain meta features of a dataset for training in a deep learning application. A method includes selecting an initial search space that defines a type of deep learning architecture representation that specifies hyperparameters for two or more neural network architectures. The method also includes applying a search strategy to the initial search space. One of the two or more neural network architectures are selected based on a result of an evaluation according to the search strategy. A new search space is generated with new hyperparameters using an evolutionary algorithm and a mutation type that defines one or more changes in the hyperparameters specified by the initial search space, and, based on the mutation type, the new hyperparameters are applied to the one of the two or more neural networks or the search strategy is applied to the new search space.