Crowd-Sourced Labeling Confidence Model for Bot Noise
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computer-implemented crowd-sourced environments, particularly in CAPTCHA systems, user identification is infeasible, leading to noisy labels and reduced accuracy due to a high fraction of non-human responses from bots, which complicates the aggregation of responses and affects the reliability of machine learning models.
Innovation Solution
A dynamic relabelling method that adaptively chooses the number of responses needed for accurate aggregation, using a confidence model to determine when to stop collecting guesses, and incorporates a control object with known labels to assess the reliability of crowd-sourced workers, thereby improving the accuracy of aggregated answers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If crowd-sourced assessors are used to label data, then the diversity of opinions and preferences is captured and cost is reduced, but the quality and reliability of labels deteriorate due to noisy responses from bots and non-qualified users
Solution Approach 1:
The patent introduces an intermediary verification system that acts as a mediator between crowd-sourced responses and the final labeled data. This system uses multiple verification mechanisms including consistency checks, confidence scoring, and iterative refinement processes to filter out noisy responses from bots while preserving valuable diverse opinions from human assessors, thus resolving the contradiction between cost efficiency and label quality
Solution Approach 2:
The patent implements feedback loops where initial labels are evaluated, and incorrect or noisy labels trigger re-verification processes. The system continuously refines label quality by incorporating feedback from verification results, adjusting the pool of assessors, and applying corrective measures to improve overall label reliability while maintaining cost-effectiveness
2Reliability
If multiple responses are collected from crowd-sourced assessors to improve accuracy, then the reliability of aggregated answers improves, but the time and computational resources required increase
Solution Approach 1:
The patent employs dynamic response collection strategies where the number of responses needed is adjusted based on real-time assessment of answer consistency and confidence levels. The system dynamically determines when sufficient responses have been collected by monitoring convergence metrics, allowing early termination when accuracy thresholds are met, thus reducing time loss while maintaining reliability
Solution Approach 2:
The system changes parameters such as the number of required responses, confidence thresholds, and aggregation methods based on the specific task characteristics and observed response quality. By adaptively adjusting these parameters, the system optimizes the balance between collecting enough responses for accuracy and minimizing time consumption
3Device complexity
If a simple majority vote method is used to aggregate responses, then the system complexity is reduced, but the accuracy deteriorates when a significant proportion of responses are from bots
Solution Approach 1:
The patent introduces intermediary verification layers between individual responses and the final aggregation process. These intermediaries include confidence scoring mechanisms, bot detection filters, and weight assignment systems that preprocess responses before aggregation, enabling more accurate results without requiring excessively complex aggregation algorithms
Solution Approach 2:
The patent segments the aggregation process into multiple stages: initial filtering of obviously incorrect responses, confidence-based weighting, iterative refinement, and final aggregation. This segmentation allows each stage to address specific aspects of accuracy improvement with relatively simple operations, avoiding the need for a single complex aggregation system
Data Source
AI summary
Disclosed are a method and a system for determining a response to a digital task in a computer-implemented crowd-sourced environment. The method comprises determining if a number of the plurality of responses to the digital task received meets a pre-determined minimum answer threshold; in response to the number of the plurality of responses to the digital task meeting the pre-determined minimum answer threshold, executing: for each of the plurality of responses generating, by the server, a confidence parameter representing a probability of an associated one of the plurality of responses being correct; ranking the plurality of responses based on the confidence parameter to determine a top response being associated with a highest confidence parameter; and in response to the highest confidence parameter being above a pre-determined minimum confidence threshold, assigning a value of the top response as a label for the digital task and terminating the digital task execution.