Label Bias Correction via Stratified Sampling and Adversarial Reassignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques fail to reliably correct label bias in training datasets, especially when the adversarial process has access to true labels, leading to inaccurate removal of biases and reduced accuracy in machine learning models.
Innovation Solution
The method involves stratified sampling to select a trusted dataset from the training data, determining the true labels of the untrusted dataset using a conditional probability distribution based on the trusted dataset, and injecting noise to reassign labels, with a confidence threshold to ensure accurate bias removal.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing techniques use adversarial process to remove bias from training data, then bias removal is attempted, but reliability and accuracy are reduced when the adversarial process has access to true labels
Solution Approach 1:
The training dataset is segmented into two distinct portions: a trusted dataset containing data samples with reliable labels, and an untrusted dataset containing data samples with potentially biased or corrupted labels. This segmentation allows the system to process and correct biases in the untrusted data by comparing against the reliable trusted data, thereby maintaining both bias removal capability and label accuracy.
Solution Approach 2:
The system introduces an intermediary confidence threshold mechanism that mediates between the adversarial bias removal process and the true labels. By calculating confidence scores for each label based on the trusted dataset and comparing against a threshold, the system determines whether to retain or correct labels, ensuring reliable bias removal without sacrificing accuracy.
2Productivity
If a small amount of trusted data is used to correct bias in corrupted data, then computational resources are reduced, but accuracy is compromised when adversarial process accesses true labels
Solution Approach 1:
Instead of processing the entire training dataset for bias correction, the system applies partial action by focusing computational resources only on the untrusted dataset portion. The trusted dataset serves as a reference and is not subjected to the computationally intensive adversarial process, thereby maintaining high processing efficiency while achieving accurate bias correction in the problematic data segments.
Solution Approach 2:
The system changes the parameter of data trustworthiness by assigning different confidence levels to different portions of the dataset. By parameterizing data samples as either trusted or untrusted, the system can apply different processing strategies - lightweight reference checking for trusted data and intensive adversarial correction for untrusted data - optimizing both efficiency and accuracy.
3Ease of manufacture
If observed labels are directly used from training data, then labeling process is simple, but bias is introduced that reduces model accuracy
Solution Approach 1:
The system performs preliminary action by pre-processing the training data to identify and separate trusted from untrusted samples before the main model training begins. This preliminary segmentation and confidence assessment ensures that bias correction is already in place before training, maintaining labeling simplicity while eliminating the source of accuracy degradation.
Solution Approach 2:
The system converts the harmful effect of biased labels into a benefit by using the trusted dataset as a reference to identify and correct biases in the untrusted dataset. The presence of labeled data, even when potentially biased, becomes useful because it can be systematically compared against the trusted portion to identify and correct errors, transforming a liability into an asset for improving model accuracy.
Data Source
AI summary
This application relates to systems and methods for automatically correcting labels in untrusted data based on a small sample of trusted data in a training database. In some examples, training data may be divided into a trusted dataset and an untrusted dataset using stratified sampling. An adversarial algorithm may be used to reassign labels in the data samples associated with the untrusted data based on a set of features in the data and labels in the trusted dataset. The untrusted dataset with the reassigned labels may then be used to train a machine learning model.