ML Pipeline Selection via Latent Factor Matrix
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In automated machine learning settings, determining the best machine learning pipeline for generating models becomes impractical due to the large number of available pipelines, as running datasets through all pipelines to assess accuracy is resource-intensive and costly.
Innovation Solution
A computer-implemented method performs cross-validation runs on dataset-pipeline combinations, builds a matrix of accuracy scores, factors them into latent factors, and selects a subset of pipelines for a new dataset to build and test models, thereby identifying the most optimal pipeline without needing to run all available pipelines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all available machine learning pipelines are run through the dataset to determine the best pipeline, then the accuracy of pipeline selection is improved, but the computational cost and resource consumption increase prohibitively
Solution Approach 1:
The system performs preliminary cross-validation runs on a subset of pipelines to build an accuracy matrix before final pipeline selection. This preliminary action allows the system to identify high-performing pipelines without evaluating all available pipelines, thereby reducing computational cost while maintaining selection accuracy.
Solution Approach 2:
The patent introduces an intermediary accuracy matrix that stores cross-validation results. This matrix serves as a mediator between the large set of available pipelines and the final selection process, allowing the system to leverage historical accuracy data to avoid re-evaluating all pipelines and instead focus computational resources on a smaller subset.
2Loss of energy
If a subset of pipelines is selected based on historical accuracy matrix, then the computational cost is reduced, but the risk of missing the optimal pipeline increases
Solution Approach 1:
The system performs cross-validation runs on more pipelines than the minimum necessary (excessive action) to build a comprehensive accuracy matrix. This partial evaluation of pipelines with sufficient redundancy ensures that the optimal pipeline is captured in the historical data, making subsequent subset selection both computationally efficient and reliable.
Solution Approach 2:
The accuracy matrix serves as a feedback mechanism that captures performance information from cross-validation runs. This feedback is used to inform subsequent pipeline selections, allowing the system to learn from historical performance data and improve the reliability of selecting optimal pipelines from subsets in future iterations.
Data Source
AI summary
Machine learning (ML) pipeline selection includes performing cross-validation runs for dataset-pipeline combinations and building a matrix of first accuracy scores, factoring the matrix of accuracy scores into pipeline latent factors and dataset latent factors, augmenting the matrix of accuracy scores by selecting a subset of ML pipelines of a plurality of ML pipelines, then, for a new dataset, running the subset of ML pipelines with the new dataset to build and test respective ML models, obtain second accuracy scores, and augment the matrix of accuracy scores with the second accuracy scores to produce an augmented matrix of accuracy scores, factoring the augmented matrix of accuracy scores into refined pipeline latent factors and refined dataset latent factors, and identifying, based on the refined pipeline latent factors and the refined dataset latent factors, ML pipeline(s), of the plurality of ML pipelines, as most optimal for model building based on the new dataset.