LDS Feature Selection and Hyperparameter Search for Nonlinear Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning methods face challenges in efficiently performing feature selection and hyperparameter optimization, particularly in non-linear models, leading to suboptimal model performance, interpretability, and resource-intensive searches.
Innovation Solution
The implementation of Limited Discrepancy Search (LDS) with a weight table for simultaneous feature selection and hyperparameter optimization, which systematically searches for improved feature sets and hyperparameters, updating a weight table during the process to optimize model performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If exhaustive search methods are used for feature selection andhyperparameter optimization, then model performance can be improved, but computational complexity and resource consumption increase significantly
Solution Approach 1:
The patent segments the feature selection andhyperparameter optimization process into discrete steps that can be systematically explored using LDS. By breaking down the search space into manageable segments with controlled discrepancies, the method achieves thorough optimization without requiring exhaustive search of all possible combinations, thus reducing computational complexity while maintaining model performance.
Solution Approach 2:
The patent performs preliminary ordering of features andhyperparameters based on their importance or impact on model performance before conducting the LDS. This preliminary action creates a structured search sequence that guides the LDS algorithm to explore the most promising configurations first, reducing the need to evaluate all possible combinations and thereby lowering computational requirements.
2Reliability
If more features are selected to improve model accuracy, then model performance increases, but model interpretability and training speed decrease
Solution Approach 1:
The patent applies partial action by selecting only the necessary subset of features andhyperparameters that provide optimal model performance rather than using all available features. The LDS algorithm with controlled discrepancy explores a limited but sufficient portion of the feature space, identifying the minimal set of features needed for good accuracy, thus maintaining model interpretability and training speed.
Solution Approach 2:
The patent systematically changes parameters (feature subsets andhyperparameter values) in a controlled manner through LDS, exploring different configurations to find the optimal balance between model accuracy and complexity. By adjusting the discrepancy level and search depth, the method identifies parameter settings that achieve high accuracy with fewer features, reducing model complexity.
3Productivity
If systematic local search with LDS is used, then computational efficiency is improved, but the search may get stuck in local minima
Solution Approach 1:
The patent introduces dynamics into the LDS process by adaptively adjusting the discrepancy level and search strategy based on progress made. The algorithm dynamically transitions between exploring nearby solutions (low discrepancy) and making larger jumps (higher discrepancy) when improvement stalls, allowing it to escape local minima while maintaining computational efficiency through targeted rather than exhaustive search.
Solution Approach 2:
The patent incorporates feedback mechanisms where the LDS algorithm monitors model performance at each step and uses this information to guide subsequent search decisions. When the search encounters local minima (no improvement), the feedback triggers adjustments in the search strategy, such as increasing discrepancy levels or reordering features, enabling the algorithm to escape suboptimal solutions and continue improving model performance efficiently.
Data Source
AI summary
A computer-implemented method for simultaneous feature selection and hyperparameter optimization of non-linear models of machine learning is provided including setting a first solution having first hyperparameters and a first set of features of a plurality of features of a training data set, initializing a weight table providing a score for each feature of the first set of features, and initializing a discrepancy. The method further includes performing a limited discrepancy search (LDS), according to an order based on the weight table, to obtain a second solution having second hyperparameters and a second set of features by swapping the first set of features and switching the first hyperparameters from the first solution with the discrepancy, while updating the weight table during LDS, comparing the second solution with the first solution, and obtaining a new solution with improved features and hyperparameters, as an optimized solution.