ML Accuracy Estimation via Cluster-Based Retraining Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face accuracy deterioration due to changes in input data distribution, leading to increased maintenance costs and difficulty in automatic retraining, especially when correct answer information is unavailable.
Innovation Solution
A performance estimation device that utilizes clustering techniques to automatically detect accuracy deterioration by matching input clusters with training clusters, generating retraining data, and adjusting boundaries based on density and persistent homology, thereby reducing manual labeling costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual labeling is performed for retraining when accuracy deteriorates, then model accuracy can be restored, but labeling costs and processing loads increase significantly
Solution Approach 1:
The system performs automatic clustering-based labeling without human intervention. The clustering unit automatically clusters input data in feature space, and the labeling unit automatically assigns labels based on cluster correspondence with training data, enabling the system to maintain itself without external manual labeling resources.
Solution Approach 2:
The system pre-establishes the correspondence relationship between clusters and labels during the clustering process. By clustering input data and mapping it to training clusters beforehand, the system prepares labeled retraining data in advance, so when accuracy deterioration is detected, retraining can immediately use pre-prepared data without costly manual labeling.
2Reliability
If retraining is performed frequently to maintain accuracy, then model performance is maintained, but processing time and computational resources increase
Solution Approach 1:
The accuracy estimation unit automatically monitors model accuracy by clustering input data and comparing it with training clusters. When accuracy deterioration is detected, the system automatically initiates retraining with pre-labeled data, reducing the time loss associated with manual accuracy monitoring and data preparation.
Solution Approach 2:
The system continuously clusters input data in the background and maintains correspondence relationships with training clusters. This preliminary clustering action ensures that when retraining is needed, labeled data is already prepared, significantly reducing the actual retraining time and computational overhead.
3Productivity
If clustering is performed to enable automatic labeling, then labeling efficiency improves, but clustering accuracy may deteriorate if not properly implemented
Solution Approach 1:
The clustering unit performs clustering in the feature space specific to the machine learning model, considering the local characteristics of the data distribution. By clustering in the transformed feature space rather than raw input space, the system achieves better clustering accuracy that respects the local structure of the data while maintaining high labeling efficiency.
Solution Approach 2:
The feature space acts as an intermediary between the input data and the clustering process. By transforming input data into feature space through the machine learning model before clustering, the system creates a more suitable representation for clustering, improving both clustering accuracy and the quality of automatic labeling.
Data Source
AI summary
A non-transitory computer-readable storage medium storing an estimation program that causes a computer to execute a process includes specifying representative points of each of training clusters that corresponds to each of labels targeted for estimation; setting boundaries between each of input clusters under a condition that a number of the input clusters and a number of the representative points coincide with each other, the input clusters being generated by clustering in a feature space for input data; acquiring estimation results for the labels with respect to the input data based on a correspondence relationship between the input clusters and the training clusters based on the boundaries; and estimating determination accuracy for the labels by using the machine learning model with respect to the input data based on the estimation results.