Label Validation System for Detecting Mislabeled Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid deployment of supervised machine learning algorithms is hindered by the lack of reliable labeled training data, as manual curation is infeasible for large datasets, and heuristic algorithms used to generate labels can introduce errors, making it difficult to identify and correct mislabeled data.
Innovation Solution
A method and system for validating labels in training data by determining a second label based on features using a model, calculating a loss between the first and second labels, and classifying records with incorrect labels, thereby generating a subgroup of mislabeled records for correction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If heuristic algorithms are used to generate labeled training data, then productivity increases and manual curation becomes feasible at scale, but the reliability of labels deteriorates due to introduced errors
Solution Approach 1:
The patent implements a feedback mechanism where the trained machine learning model generates predictions on training data, and these predictions are compared against the heuristic labels. The loss computation provides feedback to identify discrepancies, allowing the system to detect and correct mislabeled data while maintaining high productivity from heuristic labeling
Solution Approach 2:
The patent introduces a machine learning model as an intermediary between the heuristic labeling process and the final training data. This intermediary model learns from the data and can identify when heuristic labels are incorrect, thereby improving reliability without sacrificing the productivity benefits of automated labeling
2Reliability
If manual expert-labelling is used to ensure clean labeled data, then reliability of labels improves, but productivity decreases making large-scale labeling infeasible
Solution Approach 1:
Instead of manually labeling all data points, the patent applies partial manual verification only to data points where the model prediction disagrees with the heuristic label (high loss cases). This selective approach maintains high reliability for critical cases while preserving overall productivity through automated processing of the majority of data
3Productivity
If heuristic algorithms are employed for large-scale labeling, then productivity increases enabling Big-Data applications, but the difficulty of detecting and measuring incorrect labels increases
Solution Approach 1:
The system uses feedback from the trained model's predictions to automatically detect mislabeled data. By comparing model predictions with heuristic labels and computing loss values, the system identifies incorrect labels without requiring manual inspection, thus maintaining productivity while reducing detection difficulty
Solution Approach 2:
The machine learning model performs self-validation by evaluating its own training data. The model generates predictions on the training set and uses these predictions to identify potential labeling errors, enabling automatic detection and correction without external intervention
Data Source
AI summary
The present teaching relates to a method and system for validating labels of training data. A first group of data records associated with the training data are received, wherein each of the first group of data records includes a vector having at least one feature and a first label. For each of the first group of data records, a second label is determined based on the at least one feature in accordance with a first model. Thereafter, a loss based on the first label associated with the data record and the second label is obtained, and the data record having an incorrect first label is classified when the loss meets a pre-determined criterion. Upon classifying the data records, a sub-group of the first group of data records is generated, wherein each of the data records included in the sub-group has the incorrect first label.