Machine Learning Model Selection with Joint Feature and Hyperparameter Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for selecting machine learning model hyperparameters do not interact with feature selection, leading to potential mismatches and inefficiencies, and lack standardized methods for using training data to inform hyperparameter search spaces and feature selection processes.
Innovation Solution
A system that defines a hyperparameter search space based on training data using a preliminary model's parameter values, generates feature vectors for each configuration, and ranks candidate models based on performance metrics to select the best-performing model, integrating hyperparameter optimization with feature selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If hyperparameter optimization is performed separately from feature selection using conventional methods, then the hyperparameter selection process is simplified, but mismatches occur between hyperparameter configurations and feature sets leading to suboptimal model performance
Solution Approach 1:
The patent merges hyperparameter optimization and feature selection into a unified framework where they are performed jointly rather than separately. The system evaluates multiple hyperparameter configurations against multiple feature sets simultaneously, ensuring that hyperparameters are optimized for the specific feature combinations they will work with, thereby eliminating mismatches while maintaining systematic complexity through integrated evaluation.
Solution Approach 2:
The patent performs preliminary action by training a preliminary model first to establish baseline performance and identify promising hyperparameter configurations. This preliminary model training informs the subsequent joint optimization process, allowing the system to focus computational resources on configurations that have demonstrated potential, thus improving overall model performance while managing complexity through staged evaluation.
2Reliability
If a comprehensive hyperparameter search space is explored to ensure optimal model performance, then model accuracy improves, but computational time and resources increase significantly
Solution Approach 1:
The patent applies partial action by evaluating a strategically selected subset of hyperparameter configurations rather than exhaustively searching the entire search space. The system identifies and focuses on configurations that are most likely to yield optimal performance based on preliminary model results and performance thresholds, achieving high model accuracy while avoiding the prohibitive computational cost of complete enumeration.
Solution Approach 2:
The patent segments the hyperparameter search space into multiple discrete configurations that are evaluated independently and systematically. By dividing the comprehensive search space into manageable segments corresponding to specific feature set combinations, the system can parallelize evaluation and identify promising configurations more efficiently, reducing overall computational time while maintaining thoroughness.
3Productivity
If feature selection is performed without considering hyperparameter configurations, then the feature selection process is faster and simpler, but the selected features may not be optimal for the specific hyperparameter settings
Solution Approach 1:
The patent merges feature selection with hyperparameter configuration evaluation into a unified process where features are selected and evaluated in the context of specific hyperparameter settings. This joint evaluation ensures that feature relevance is assessed based on actual model performance with given hyperparameters, achieving precise feature selection while maintaining productivity through integrated and parallelized evaluation of multiple configurations simultaneously.
Data Source
AI summary
Systems and methods for selecting machine learning features using iterative batch feature reduction. In some aspects, the system receives training data intended for generating a machine learning model. Based on a preliminary model trained on the training data, the system defines a hyperparameter search space to generate a set of hyperparameter configurations. For each hyperparameter configuration, the system generates a feature vector by executing a feature selection method. Based on the set of feature vectors and the training data, the system generates a set of candidate models corresponding to the set of hyperparameter configurations. The system ranks the set of candidate models based on a performance metric to select the machine learning model from the set of candidate models.