Hyperparameter Estimation via Iterative Cross-Entropy Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining hyperparameters in machine learning classifiers, such as the cross-entropy method, are not suitable due to the large size of the search space and the difficulty in parameterizing hyperparameter samples, which are not classical probability density functions.
Innovation Solution
A method that iteratively selects and updates hyperparameter vectors by choosing the best-performing vector from random samples, using a weighting function based on Euclidean distance to guide the next iteration and restrict the sample space, allowing for efficient hyperparameter estimation even when hyperparameters are not continuous.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the cross-entropy method is used for hyperparameter estimation, then sampling efficiency is improved, but the method becomes unsuitable due to the large search space and difficulty in parameterizing hyperparameter samples
Solution Approach 1:
The patent segments the hyperparameter search space by dividing it into multiple dimensions, where each dimension corresponds to a specific hyperparameter. This segmentation allows the application of cross-entropy sampling to each dimension independently, making the large search space manageable and parameterizable while maintaining sampling efficiency.
2Reliability
If hyperparameters are selected from a large search space, then the chance of finding optimal parameters increases, but the computational complexity and time required increase significantly
Solution Approach 1:
The patent applies preliminary action by using cross-entropy sampling to pre-identify promising hyperparameter regions before full optimization. The method performs preliminary sampling to estimate important hyperparameter values, which then guide the subsequent optimization process, reducing the time needed to reach optimal hyperparameters while maintaining reliability.
Solution Approach 2:
The patent implements feedback by using the results from cross-entropy sampling to update and refine the search strategy in subsequent iterations. The sampling distribution is updated based on observed performance, creating a feedback loop that progressively narrows down to optimal hyperparameters more efficiently than exhaustive search.
3Adaptability or versatility
If random sampling is used to explore hyperparameter space, then coverage is improved, but convergence to optimal values becomes slow
Solution Approach 1:
The patent applies dynamics by making the sampling distribution adaptive rather than static. The cross-entropy method dynamically updates the sampling distribution based on observed performance, allowing the sampling process to evolve from broad coverage to focused exploration of promising regions, thereby improving convergence speed while maintaining adequate space coverage.
Solution Approach 2:
The patent uses parameter changes by modifying the sampling distribution parameters (means and variances) based on observed hyperparameter performance. This allows the sampling process to transition from uniform exploration to targeted exploitation, improving convergence speed while maintaining adaptability to the underlying performance landscape.
Data Source
AI summary
A method of determining hyperparameters (HP) of a classifier (1) in a machine learning system (10) iteratively produces an estimate of a target hyperparameter vector. The method comprises the steps of selecting from the random sample the hyperparameter vector producing the best result in the present and any previous iterations, and updating the estimate of the target hyperparameter vector by using said selected hyperparameter vector. The random sample may be restricted by using the hyperparameter vector producing the best result in the present and any previous iterations.