Meta-Model Performance Prediction for Machine Learning Model Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models often struggle to achieve optimal performance across heterogeneous and high-dimensional data sets due to variations in data quality and characteristics, necessitating multiple models for different data subsets, making model selection for new data challenging and resource-intensive.
Innovation Solution
Development of meta-models that utilize meta-features generated through optimization algorithms to predict the performance of candidate models on input data subsets, reducing the need for extensive experimentation by recommending optimal models and providing data quality and model enhancement recommendations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple machine learning models are trained to handle different data subsets, then model performance on specific data subsets is improved, but model selection complexity and resource consumption increase
Solution Approach 1:
The patent introduces an intermediary system consisting of meta-models and feature extraction mechanisms that mediate between the diverse data subsets and multiple candidate machine learning models. This intermediary layer automatically analyzes data characteristics and recommends appropriate models, eliminating the need for complex manual model selection processes while maintaining high performance on specific data subsets.
2Reliability
If multiple machine learning models are trained to handle different data subsets, then model performance on specific data subsets is improved, but computing and resource usage increase
Solution Approach 1:
The patent applies preliminary action by pre-training multiple candidate machine learning models on historical data subsets and storing them in a model repository. When new data arrives, the system only needs to extract features and query the pre-trained models rather than training models from scratch, significantly reducing computing resource usage while maintaining the ability to handle different data subsets effectively.
Solution Approach 2:
The system creates copies of trained models for different data subsets and stores them in a repository. Instead of retraining models each time, the system copies and applies the appropriate pre-trained model to new data based on feature extraction, reducing computational overhead while preserving model performance on specific data characteristics.
3Measurement precision
If extensive experimentation is conducted to select the best model for new data, then model selection accuracy is improved, but time consumption increases
Solution Approach 1:
The patent extracts essential features from new data that characterize the data subset, and uses these extracted features to query the model repository for the most suitable pre-trained model. This extraction-based approach eliminates the need for extensive experimentation while maintaining accurate model selection, as the system only needs to identify key data characteristics rather than test multiple models exhaustively.
Data Source
AI summary
Result quality metrics of a set of machine learning tasks conducted on various record groups using a plurality of machine learning models are obtained. Based on applying an algorithm to the record groups, respective sets of intermediary results corresponding to records of the groups are obtained. A meta-model for predicting result quality metrics for respective record-group-and-model combinations is trained using a training data set which includes statistical features obtained from the intermediary results. The trained meta-model is stored.