Labeled Data Relabeling Using Multi-Model Uncertainty Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inaccurate labeling of training data leads to biased machine learning models, causing inaccurate predictions and negative impacts on products and user experiences, with data uncertainty largely underutilized and knowledge uncertainty being addressed through active learning techniques.
Innovation Solution
Utilize query by committee (QBC) to quantify knowledge uncertainty and estimate data uncertainty by training multiple models, determining label uncertainty scores to identify mislabeled data, and submitting them to domain experts for correction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple machine learning models are trained to quantify knowledge uncertainty, then the ability to identify mislabeled data improves, but the computational complexity and time required increases
Solution Approach 1:
The system pre-trains multiple machine learning models with different architectures and configurations before they are needed for uncertainty quantification. These pre-trained models are ready to be deployed immediately when labeled data needs to be evaluated, eliminating the time-consuming model training step during the actual data processing workflow.
Solution Approach 2:
The evaluation process is divided into separate stages: first, multiple pre-trained models independently process the labeled data to generate predictions; second, the predictions are aggregated to calculate entropy scores and determine label uncertainty. This segmentation allows parallel processing of different models and optimizes the overall computation efficiency.
2Reliability
If multiple machine learning models are used to process labeled data, then data uncertainty detection improves, but the device complexity increases
Solution Approach 1:
Multiple machine learning models with different architectures (e.g., decision trees, neural networks, support vector machines) are employed, each serving the universal function of classifying labeled data. Their diverse algorithmic approaches provide complementary perspectives on data uncertainty, improving detection reliability without requiring each model to be highly specialized.
Solution Approach 2:
The system implements a feedback mechanism where the predictions from multiple models are aggregated through entropy calculation. This feedback loop allows the system to identify patterns of disagreement among models, which indicates high uncertainty and potential mislabeling, thereby improving detection reliability through collective model wisdom.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems, methods, and software can be used to determine whether to re-label a labeled data. In some aspects, a method includes: obtaining, by an electronic device, a set of labeled data, wherein each of the labeled data comprises a feature vector and a label; for each labeled data in the set of the labeled data: processing the labeled data to obtain a plurality of classification results by using a plurality of machine learning models, wherein each of the plurality of classification results is obtained by using a different machine learning model in the plurality of machine learning models to process the feature vector of the labeled data; and determining a label uncertainty score of the labeled data based on a difference between an average entropy score and an adjustment score; and determining, whether to re-label one or more labeled data in the set of labeled data based on the label uncertainty scores.