Iterative Feature Selection for Sparse Machine Learning Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing supervised learning models often become overfit, sensitive to noise, and difficult to interpret when trained on datasets with a large number of features, leading to resource-intensive processing requirements.
Innovation Solution
The method involves iteratively training models based on a reduced feature set by identifying and removing features with low contribution to predictions, using techniques like SHAP values, and storing data in sparse formats like CSR to efficiently manipulate and retrain models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large number of features are used to train the model, then the model may achieve better predictive performance, but the model becomes overfit, sensitive to noise, and difficult to interpret
Solution Approach 1:
The patent extracts and removes features that contribute minimally to predictive performance. By calculating feature contribution metrics (such as SHAP values) and iteratively removing the least important features, the system achieves model simplification while preserving predictive accuracy. This directly addresses the contradiction by taking out unnecessary features that increase complexity without providing proportional performance benefits.
Solution Approach 2:
The patent changes the parameter of feature dimensionality by iteratively adjusting the number of features in the training set. Through multiple iterations of model training, feature contribution calculation, and feature removal, the system transforms the feature set from high-dimensional to an optimized lower-dimensional representation that maintains reliability while reducing complexity.
2Loss of information
If a large number of features are used to train the model, then the model may capture more information, but the processing requirements increase significantly
Solution Approach 1:
The patent applies partial action by training the model iteratively on progressively reduced feature sets rather than using all features from the beginning. Each iteration uses a subset of features, and the process continues until the desired balance between information retention and processing efficiency is achieved. This avoids the excessive processing requirements of using all features while preserving the most important information.
Solution Approach 2:
The system changes the parameter of feature set size across multiple iterations, starting with a complete feature set and progressively reducing it. This parameter transformation allows the system to identify and retain only the features that provide the most valuable information, thereby reducing processing requirements while minimizing information loss.
3Device complexity
If existing techniques are used to limit features, then the resulting model is smaller, but the processing requirements of the feature selection techniques outweigh the resource savings
Solution Approach 1:
The patent implements feedback by calculating feature contribution metrics after each model training iteration and using this information to guide the next feature removal decision. The SHAP value calculations provide feedback on which features are most important, allowing the system to make informed decisions about feature removal. This feedback mechanism ensures that feature selection processing is focused and efficient, avoiding unnecessary computations while achieving model simplification.
Solution Approach 2:
The system performs preliminary feature contribution analysis before final model deployment. By calculating feature importance metrics and removing low-contributing features in advance, the system reduces the processing burden of the final model while ensuring that only necessary features are retained. This preliminary action prevents wasteful processing in later stages.
Data Source
AI summary
Systems and methods include determination of a first plurality of sets of data, each including values associated with respective ones of a first plurality of features, partial training of a first machine-learning model based on the first plurality of sets of data, determination of one or more of the first plurality of features to remove based on the partially-trained first machine-learning model, removal of the one or more of the first plurality of features to generate a second plurality of sets of data, partial training of a second machine-learning model based on the second plurality of sets of data, determination that a performance of the partially-trained second machine-learning model is less than a threshold, addition, in response to the determination, of the one or more of the first plurality of features to the second plurality of sets of data, and training of the partially-trained first machine-learning model based on the first plurality of sets of data.