Active Learning via Miss-Prediction Probability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current active learning methods, such as those based on entropy values, are inefficient in selecting data samples for annotation, leading to slow performance enhancement and limited application scope, particularly in regression tasks and requiring significant human and time costs.
Innovation Solution
A machine learning method that selects data samples based on miss-prediction probabilities rather than uncertainty, using a miss-prediction probability calculation model to identify samples the model is likely to be wrong on, thereby reducing annotation costs and expanding the method's applicability beyond classification tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If active learning based on entropy values is used to select data samples, then annotation costs are reduced, but the accuracy of data sample selection is insufficient leading to slow model performance enhancement
Solution Approach 1:
The patent changes the selection criterion from entropy-based uncertainty to miss-prediction probability. This parameter change enables more accurate identification of samples that the model is likely to get wrong, thereby improving both selection accuracy and model performance enhancement while still reducing annotation costs.
Solution Approach 2:
The patent replaces the entropy-based selection mechanism with a miss-prediction probability calculation mechanism. This substitution uses a separate calculation model to evaluate the likelihood of wrong predictions, providing a more direct and accurate measure for selecting samples that will most benefit from annotation.
2Loss of time
If entropy-based active learning is applied, then annotation costs are reduced, but the application scope is limited to classification tasks only
Solution Approach 1:
The patent creates a universal active learning method based on miss-prediction probability that can be applied to both classification and regression tasks. The miss-prediction probability calculation model is task-agnostic and can evaluate sample selection needs across different machine learning problem types, thereby expanding adaptability while maintaining annotation cost efficiency.
3Reliability
If all data samples are annotated to construct training dataset, then model performance is maximized, but human cost and time cost are significantly consumed
Solution Approach 1:
The patent extracts only the most valuable subset of data samples for annotation based on miss-prediction probability. By identifying and selecting only those samples that the model is likely to get wrong, the method extracts the essential information needed for model improvement while avoiding the time cost of annotating all samples.
Solution Approach 2:
The patent applies partial action by annotating only a portion of the data samples rather than all samples. The miss-prediction probability mechanism identifies the critical subset of samples that, when annotated, provide the maximum performance improvement per unit of annotation effort.
Data Source
AI summary
A machine learning method that may reduce an annotation cost and may improve performance of a target model is provided. Some embodiments of the present disclosure may provide a machine learning method performed by a computing device, including: acquiring a training dataset of a first model including a plurality of data samples to which label information is not given; calculating a miss-prediction probability of the first model on the plurality of data samples; configuring a first data sample group by selecting at least one data sample from the plurality of data samples based on the calculated miss-prediction probability; acquiring first label information on the first data sample group; and performing first learning on the first model by using the first data sample group and the first label information.