Crowdsourced Training Data Quality via Assessor Consistency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Crowdsourced training data for machine-learning algorithms is often noisy due to non-professional assessors and fraudulent activity, leading to decreased quality and increased costs in verifying accuracy.

Innovation Solution

A method that identifies sets of assessors based on a consistency metric, which maximizes the posteriori probability of correct results, allowing for the selection of assessors with optimal quality scores for subsequent tasks without the need for additional control tasks, thereby reducing noise and improving data quality.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If control tasks are used to assess assessor quality, then the quality of training data is improved, but the cost of generating training data increases significantly

Engineering Contradiction:

Improvequality of training dataVSAvoidcost of generating training data

Core Design Contradiction:

ReliabilityVSEase of manufacture

Solution Approach 1:

The patent extracts the quality assessment function from separate control tasks and integrates it into the main task execution process. By using peer consistency evaluation among multiple assessors completing the same task, the system eliminates the need for dedicated control tasks while maintaining quality control, thus reducing costs without sacrificing data quality

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system uses assessors themselves to evaluate each other's work through peer consistency evaluation. Multiple assessors complete the same task independently, and their agreements serve as the quality metric. This self-evaluation mechanism eliminates the need for external quality control tasks and reduces overall system costs

Inventive Principle:

Principle #25Self-service

2Reliability

If control tasks are used to detect fraudulent labelling, then the quality of training data is improved, but the time required for data generation increases

Engineering Contradiction:

Improvequality of training dataVSAvoidtime required for data generation

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent merges quality control and data generation into a single process. Multiple assessors complete tasks simultaneously, and quality assessment through peer consistency evaluation occurs as part of the same workflow rather than as a separate subsequent step. This integration eliminates the time penalty associated with separate control task execution

Inventive Principle:

Principle #5Merging (Combining)

3Reliability

If assessors are selected based solely on quality scores from control tasks, then the quality of training data is improved, but fraudulent assessors can still execute control tasks faithfully while neglecting other tasks

Engineering Contradiction:

Improvequality of training dataVSAvoidaccuracy of assessor selection

Core Design Contradiction:

ReliabilityVSManufacturing precision

Solution Approach 1:

The patent segments the evaluation of assessor quality into multiple dimensions: consistency with majority results across tasks and adherence to instructions. By evaluating assessors on both peer consistency and instruction following, the system prevents fraudsters from manipulating a single control task metric while maintaining accurate identification of reliable assessors

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system implements feedback through peer consistency evaluation where each assessor's work is continuously compared against the majority of other assessors. This ongoing feedback mechanism detects fraudulent behavior patterns as they occur, allowing for real-time identification and exclusion of unreliable assessors from future tasks

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS20220292396A1Method and system for generating training data for a machine-learning algorithm

Publication Date: 2022.09.15 Y E HUB ARMENIA LLC

AI summary

A method and a system for generating training data for an MLA are provided. The method comprises: retrieving assessor data associated with a plurality of assessors, the assessor data including data indicative of a plurality of results responsive to a given digital task having been submitted to the plurality of assessors; based on the plurality of results, determining at least one set of assessors in the plurality of assessors, such that a consistency metric amongst results provided by the at least one set of assessors for the given digital task is maximized, transmitting a subsequent digital task to respective electronic devices associated with the at least one set of assessors; and generating the training data for the computer-executable MLA including data generated in response to respective ones of the at least one set of assessors completing the subsequent digital task.