Binary Classification Device Annotator Bias Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In binary classification, training datasets obtained through shared annotation by multiple annotators lack consistency due to individual biases, leading to inconsistent reliability additions.
Innovation Solution
A binary classification device that includes a reliability addition distribution generator, a bias corrector, and a corrected reliability output unit, which calculates and corrects annotator biases by referencing a reference reliability addition distribution to produce a consistent training dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple annotators perform annotation in a shared manner to increase productivity, then the quantity of annotated data increases, but the consistency of the training data set deteriorates due to individual biases
Solution Approach 1:
The patent introduces a reliability addition distribution as an intermediary mechanism that mediates between multiple annotators' subjective judgments. This distribution serves as a standardized reference framework that translates individual annotator biases into a common statistical language, allowing consistent aggregation of annotations while preserving productivity benefits of multiple annotators
Solution Approach 2:
The patent transforms the quality assessment from subjective binary judgments to objective statistical parameters by calculating reliability addition distributions. This parameter transformation converts inconsistent qualitative annotations into quantifiable statistical measures that can be consistently processed and aggregated
2Reliability
If annotator biases are corrected by referencing a reference reliability addition distribution, then data consistency improves, but the device complexity increases due to additional processing components
Solution Approach 1:
The system performs self-calibration by automatically generating reference reliability addition distributions from annotated data without requiring external intervention or manual tuning. The bias correction mechanism uses the reference distribution to automatically adjust individual annotator reliabilities, making the system self-regulating and reducing operational complexity
Data Source
AI summary
A binary classification device according to the present disclosure technology includes a reliability addition distribution generator to calculate, for each of annotators, a reliability addition histogram regarding a reliability added by the annotator, a bias corrector to correct, by referring to a reference reliability addition distribution, the reliability addition histogram to a corrected reliability addition histogram having the same characteristic as a characteristic of the reference reliability addition distribution, and a corrected reliability output to correct the reliability added by the annotator by referring to the corrected reliability addition histogram.