Machine Learning Classifier Consensus for Label Quality Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of manually generating ground truth labels for AI projects is tedious, time-consuming, and potentially inaccurate due to poor guidelines, poor grader training, or carelessness, leading to low quality labels.
Innovation Solution
A system and method that uses multiple machine learned classifiers to generate expert labels and determine an expert consensus label, comparing it to the ground truth label to identify clean labels or require reassessment, thereby improving training data quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual generation of ground truth labels is used, then label quality can be maintained through human judgment, but the process becomes tedious, time-consuming, and expensive
Solution Approach 1:
The patent introduces machine learned classifiers as intermediary experts that generate expert labels to mediate between raw data and final ground truth labels. These classifiers act as intermediate processing layers that pre-evaluate samples, reducing the burden on human graders while maintaining quality through automated consensus mechanisms.
Solution Approach 2:
The system enables self-service by allowing machine learned classifiers to automatically generate and evaluate expert labels without continuous human intervention. The automated consensus mechanism allows the system to self-regulate label quality by comparing multiple expert classifications and identifying clean samples independently.
2Productivity
If multiple machine learned classifiers are used to generate expert labels, then automated label generation is achieved, but system complexity increases
Solution Approach 1:
The patent segments the label generation task into multiple independent machine learned classifiers, each specializing in different aspects of sample evaluation. This segmentation allows parallel processing and distribution of complexity across multiple specialized components rather than one monolithic system.
Solution Approach 2:
The patent merges the outputs of multiple machine learned classifiers through an automated consensus mechanism that combines individual expert labels into a unified ground truth determination. This merging process integrates multiple specialized systems into a cohesive decision-making framework.
3Measurement precision
If expert consensus labeling is implemented, then label accuracy is improved through multiple classifications, but processing time increases
Solution Approach 1:
The patent applies partial action by implementing expert consensus labeling selectively rather than universally. The system identifies and processes only those samples that require expert review, while automatically accepting samples with clear, unambiguous classifications, thus avoiding excessive processing of already-certain cases.
Data Source
AI summary
A method includes generating, using at least one processor of an electronic device, a plurality of expert labels for a sample using a plurality of machine learned classifiers. The method also includes determining, using the at least one processor, an expert consensus label among the plurality of expert labels. The method further includes comparing, using the at least one processor, the expert consensus label to a ground truth label associated with the sample in response to determining that a consensus is found among the plurality of machine learned classifiers. The method also includes identifying, using the at least one processor, the ground truth label as a clean label in response to determining that the expert consensus label and the ground truth label match. In addition, the method includes identifying, using the at least one processor, the ground truth label for reassessment in response to determining that the expert consensus label and the ground truth label do not match.