Hyperparameter Optimization via Weighted Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current hyperparameter optimization methods, such as grid search and random search, are inefficient in terms of computing cost and time, and rely heavily on expert experience, making it difficult to find optimal hyperparameter settings for machine learning models.
Innovation Solution
A method involving calculating evaluation scores for hyperparameter samples, performing weighted clustering, and constructing an optimal hyperparameter sample set based on these scores to efficiently optimize hyperparameters, reducing computing costs and improving model performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If grid search technique is used to determine optimal combination of hyperparameters, then optimal hyperparameter combination can be found, but computing cost and time cost become excessively large
Solution Approach 1:
The patent segments the hyperparameter optimization process into multiple iterations. In each iteration, only a subset of hyperparameters is updated based on clustering results from previous iterations, rather than searching all hyperparameters simultaneously. This segmentation reduces the search space in each iteration while maintaining overall optimization accuracy.
Solution Approach 2:
The patent performs preliminary clustering of hyperparameter samples based on evaluation scores before constructing the next sample set. This preliminary action identifies promising hyperparameter regions in advance, allowing the optimization to focus on specific areas rather than exhaustively searching the entire space, thus reducing time cost while maintaining reliability.
2Reliability
If grid search technique is used to determine optimal combination ofhyperparameters, then optimalhyperparameter combination can be found, but computing cost becomes excessively large
Solution Approach 1:
The patent applies partial action by updating only a subset ofhyperparameters in each iteration rather than allhyperparameters. The clustering-based selection identifies whichhyperparameters need updating, reducing the number of model training operations and associated computing costs while still achieving optimal hyperparameter combinations through multiple iterations.
Solution Approach 2:
The patent changes the parameter representation by using clustering results to guide hyperparameter updates. Instead of systematically varying all parameters as in grid search, the method transforms the optimization into a clustering-based parameter selection process, reducing the effective parameter search space and computing requirements.
3Productivity
If random search technique is used to find optimal combination ofhyperparameters, then computing cost is reduced, but it is virtually difficult to produce optimal combination ofhyperparameters and result may vary every time
Solution Approach 1:
The patent implements feedback by using evaluation scores from model training to guide the clustering process. The clustering algorithm uses these scores as weights to identify promisinghyperparameter regions, creating a feedback loop where previous iteration results directly influence the next search direction. This feedback mechanism ensures convergence toward optimal solutions while maintaining efficiency.
Solution Approach 2:
The patent introduces dynamics by making the hyperparameter search space adaptive across iterations. The clustering results dynamically reshape whichhyperparameters are updated and how they are sampled in subsequent iterations, transforming the static random search into a dynamic, adaptive process that converges toward optimal solutions with consistent results.
Data Source
AI summary
A hyperparameter optimization method performed by a hyperparameter optimization apparatus to optimize hyperparameters of a model includes calculating an evaluation score for each of a plurality of hyperparameter samples constituting a first hyperparameter sample set by applying each of the hyperparameter samples to a target model, performing weighted clustering of the hyperparameter samples by using the calculated evaluation scores as weights and constructing a second hyperparameter sample set based on the clustering result.