Unsupervised ML Model Configuration Selection via Synthetic Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Machine Learning (ML) model configuration evaluation frameworks lack the ability to automatically and unsupervisedly select suitable model configurations for specific tasks, such as anomaly detection, without true labels, and fail to provide comprehensive feedback on model usability and performance.
Innovation Solution
A system and method that utilize unsupervised ML algorithms and synthetic labeling to evaluate and rank multiple model configurations by clustering unlabeled training data, training models with synthetic labels, and iteratively adjusting parameters based on scoring to achieve optimal performance, enabling selection of the best-suited model configuration for a given task.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If unsupervised ML algorithms are used to evaluate model configurations without labeled data, then the system can operate autonomously without manual labeling, but the precision of model evaluation and selection is reduced due to lack of true labels
Solution Approach 1:
The patent introduces synthetic labels as an intermediary between unlabeled data and model evaluation. These synthetic labels, generated through clustering algorithms, serve as a proxy for true labels, enabling automated model configuration selection while maintaining evaluation precision. The synthetic labels mediate the evaluation process by providing reference points for comparing model predictions without requiring manual annotation.
Solution Approach 2:
The system implements feedback loops where model predictions are continuously evaluated against synthetic labels, and performance metrics are fed back to guide further model configuration adjustments. This feedback mechanism enables iterative improvement of model configurations based on evaluation results, maintaining precision through continuous optimization even without true labeled data.
2Adaptability or versatility
If multiple model configurations are evaluated and ranked, then the selection of suitable models for specific tasks is improved, but the computational complexity and time required for evaluation increases
Solution Approach 1:
The patent performs preliminary clustering of unlabeled data to generate synthetic labels before model evaluation begins. This preliminary action prepares the evaluation framework in advance, creating reference synthetic labels that can be reused across multiple model configurations. By pre-processing the data and establishing synthetic ground truth beforehand, the system reduces the time required for evaluating each subsequent model configuration.
Solution Approach 2:
The evaluation process is segmented into distinct phases: synthetic label generation, model training, prediction evaluation, and configuration ranking. This segmentation allows parallel processing of multiple model configurations against the same synthetic labels, reducing overall evaluation time while maintaining comprehensive adaptability assessment across different task requirements.
3Ease of operation
If synthetic labeling is used to label unlabeled training data, then the framework can provide automated feedback on model usability, but the reliability of evaluation is reduced due to potential inaccuracies in synthetic labels
Solution Approach 1:
The system dynamically adjusts the clustering parameters and synthetic label generation process based on the characteristics of the unlabeled data and the specific task requirements. This dynamic approach allows the synthetic labels to adapt to different data distributions and task contexts, improving reliability by optimizing the synthetic labeling process for each evaluation scenario rather than using fixed labeling rules.
Solution Approach 2:
The patent varies parameters in the synthetic label generation process, such as clustering algorithm parameters, number of clusters, and distance metrics, to optimize evaluation reliability. By systematically adjusting these parameters and selecting the configuration that produces the most consistent and task-appropriate synthetic labels, the system enhances evaluation reliability while maintaining automated operation.
Data Source
AI summary
A model configuration selection system, the model configuration selection system comprising a processing circuitry configured to: (A) obtain: (a) one or more model configurations, each model configuration includes a set of parameters utilized to generate respective models, and (b) a training data-set comprising a plurality of unlabeled records, each unlabeled record including a collection of features describing a given state of a physical entity; (B) cluster the training data-set into two or more training data-set clusters using a clustering algorithm; (C) label (a) the unlabeled records of a subset of the training data-set clusters with a synthetic normal label, giving rise to a normal training data-set, and (b) the unlabeled records of the training data-set clusters not included in the subset with a synthetic abnormal label; (D) train, for each model configuration, using the normal training data-set, a corresponding model utilizing the corresponding set of parameters, each model capable of receiving the unlabeled records, and determining, for each of the unlabeled records, a corresponding normal label or abnormal label, wherein the normal label being indicative of conformity of the respective unlabeled record with an allowed state of the physical entity and the abnormal label being indicative of conformity of the respective unlabeled record with a disallowed state of the physical entity; (E) determine, for each model, a score, associated with an ability of the corresponding model to determine labels to the unlabeled records of the training data-set in accordance with the synthetic normal labels and with the synthetic abnormal labels; and (F) perform an action, based on the scores.