Meta-Learning for Unsupervised ML Pipeline Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated machine learning (AutoML) faces challenges in finding an optimal machine learning pipeline for unsupervised environments where no data labels are available to tune hyperparameters, making it difficult to configure ML models effectively.
Innovation Solution
A meta-learning-based approach is used to train a supervised model by assessing the performance of unsupervised machine learning pipelines against labeled data sets, associating performance with meta-features, and identifying suitable pipelines for processing unknown input data sets using rank correlation and receiver operating characteristics (ROC) curves.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional text matching or fixed pipelines are used for pipeline selection, then the process is simple, but the performance on unsupervised data sets is poor
Solution Approach 1:
The system performs preliminary assessment of multiple unsupervised ML pipelines against multiple labeled data sets before actual deployment. Performance metrics and meta-features are pre-computed and stored, creating a training corpus that enables the meta-learning model to make informed pipeline selections without real-time trial and error
Solution Approach 2:
A supervised meta-learning model is introduced as an intermediary between the unsupervised ML pipelines and the input data sets. This model learns from pre-assessed performance data and meta-features to predict optimal pipeline selections, avoiding direct trial-and-error evaluation on target data
2Extent of automation
If supervised training data is used to train a meta-learning model, then automated pipeline identification is enabled, but the approach requires labeled data that is not available in unsupervised environments
Solution Approach 1:
The problem is segmented into two distinct phases: (1) offline training phase where supervised meta-learning models are trained using pre-assessed performance data from multiple pipelines on multiple labeled data sets, and (2) online deployment phase where the trained model automatically identifies pipelines for unsupervised input data sets without requiring labels
Solution Approach 2:
The system creates a surrogate training environment using multiple labeled data sets that mimics the characteristics of target unsupervised environments. The meta-learning model learns from this copied training data to generalize pipeline selection capabilities to the actual unsupervised deployment scenario
3Loss of information
If multiple unsupervised ML pipelines are assessed against multiple data sets, then comprehensive performance data is obtained, but the computational cost and time increase significantly
Solution Approach 1:
Performance assessment of multiple unsupervised ML pipelines against multiple labeled data sets is performed in advance during an offline training phase. The results are stored as pre-computed performance metrics and meta-features, eliminating the need for repeated real-time assessments when deploying to new unsupervised data sets
Solution Approach 2:
The system assesses pipelines against multiple data sets (excessive action) during the training phase to ensure comprehensive coverage and robust model learning. However, this extensive assessment is performed only once offline, while actual deployment requires minimal additional assessment time
Data Source
AI summary
Computer hardware and/or software that performs the following operations: (i) assessing a performance of a plurality of unsupervised machine learning pipelines against a plurality of data sets; (ii) associating the performance with meta-features corresponding to respective pipeline/data set combinations; (iii) training a supervised meta-learning model using the associated performance and meta-features as training data; and (iv) utilizing the trained model to identify one or more pipelines for processing an input data set.