Resource-Aware Hyperparameter Optimization via Constrained Bayesian Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning model optimization methods face challenges in accurately optimizing hyper-parameters under resource constraints, leading to inefficient resource consumption and suboptimal performance.
Innovation Solution
A resource-aware automatic machine learning system that employs a hybrid methodology combining modified Bayesian optimization and genetic algorithms to efficiently optimize hyper-parameters within constrained search spaces, ensuring optimal performance while adhering to resource limitations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing machine learning optimization methods are used, then hyper-parameter optimization can be performed, but resource consumption is inefficient and performance is suboptimal
Solution Approach 1:
The patent modifies the Bayesian optimization process by introducing constraint-aware acquisition functions that incorporate resource constraints as parameters. This changes the optimization approach from unconstrained to constrained parameter search, enabling efficient resource utilization while maintaining high optimization performance through adaptive hyper-parameter selection under resource limits
Solution Approach 2:
The system implements feedback mechanisms where resource consumption metrics are continuously monitored and fed back into the optimization process. The constraint-aware acquisition function uses previous resource usage information to guide future hyper-parameter selections, creating a closed-loop system that adapts to resource constraints and improves both efficiency and performance
2Use of energy by moving object
If constrained optimization is implemented, then resource consumption is reduced, but optimization complexity increases
Solution Approach 1:
The patent creates a universal optimization framework where the constraint-aware acquisition function can handle multiple types of resource constraints (computational resources, memory, time) within a single unified process. This multi-functional approach reduces overall system complexity by consolidating constraint handling mechanisms while maintaining effective resource management across different optimization scenarios
3Productivity
If hyper-parameter search space is constrained, then resource efficiency improves, but the number of possible configurations decreases
Solution Approach 1:
The system dynamically adjusts the search space boundaries and constraint parameters based on the specific optimization problem and resource availability. The constraint-aware acquisition function adapts the hyper-parameter search space in real-time, maintaining diversity in configurations that are most likely to yield high performance while respecting current resource constraints, thus balancing productivity with adaptability
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure relates to a system, a method, and a product for optimizing hyper-parameters for generation and execution of a machine-learning model under constraints. The system includes a memory storing instructions and a processor in communication with the memory. When executed by the processor, the instructions cause the processor to obtain input data and an initial hyper-parameter set; for an iteration, to build a machine learning model based on the hyper-parameter set, evaluate the machine learning model based on the target data to obtain a performance metrics set, and determine whether the performance metrics set satisfies the stopping criteria set. If yes, the instructions cause the processor to perform an exploitation process to obtain an optimal hyper-parameter set, and exit the iteration; if no, perform an exploration process to obtain a next hyper-parameter set, and perform a next iteration with using the next hyper-parameter set as the hyper-parameter set.