Adaptive Probability Thresholds for Machine-Learning Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data labeling methods, whether manual or using machine learning models, are inefficient and prone to inconsistencies due to subjective factors and dynamic data distribution changes, affecting precision and recall rates.
Innovation Solution
A method for dynamically adjusting probability thresholds in machine learning models to adapt to data distribution changes, ensuring confidence and precision by maximizing the number of positives while maintaining a confidence threshold, thereby improving labeling efficiency and consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual audit method is used for labeling, then labeling accuracy can be maintained, but labeling efficiency is low and cost is high
Solution Approach 1:
The system enables automatic labeling through machine learning models that process data independently without requiring manual intervention for every sample. The model serves itself by automatically classifying samples and only requiring manual review when confidence thresholds are not met, thereby improving efficiency while maintaining accuracy.
2Productivity
If machine learning model is used for automatic labeling, then labeling efficiency is improved, but labeling result consistency is affected by subjective factors and data distribution changes
Solution Approach 1:
The system dynamically adjusts the confidence threshold based on data distribution characteristics and model performance. Instead of using a fixed threshold, the threshold adapts to changing conditions, ensuring consistent labeling results even when data distribution shifts occur. This dynamic adjustment mechanism maintains reliability while preserving high labeling efficiency.
Solution Approach 2:
The system implements feedback mechanisms where manual audit results are used to retrain and refine the machine learning model. This continuous feedback loop improves model consistency over time and reduces the impact of subjective factors, thereby enhancing labeling result reliability while maintaining high efficiency.
3Device complexity
If fixed probability threshold is used, then classification process is simple, but precision and recall rates are affected by data distribution changes
Solution Approach 1:
The system changes the probability threshold parameter dynamically based on data distribution characteristics. By adjusting this key parameter in response to changing conditions, the system maintains high precision and recall rates without requiring complex classification processes. The threshold adaptation is achieved through analyzing data distribution statistics and model confidence levels.
Data Source
AI summary
The present disclosure relates to a data processing method, an electronic device, a storage medium and a program product, and relates to the field of data processing. The data processing method includes: processing each sample in a first sample set by using a first machine learning model to obtain a prediction probability that each sample is classified into each category of one or more categories; and determining, for the each category, a probability threshold corresponding to the category to maximize a number of positives of the category, wherein a confidence corresponding to the category is not lower than a confidence threshold, the probability threshold is for determining a category to which the each sample pertains based on a prediction probability of the each sample, and the confidence corresponding to the category is a confidence at which an actual precision of classification based on the probability threshold meets a precision condition.