Crowdsourcing Reliability Weighting for Accurate Task Results
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for determining task results through crowdsourcing fail to accurately account for the varying review abilities of workers, leading to unreliable results due to equal weighting of all reviewers, regardless of their abilities, and inability to adapt to different types of tasks or lack of prior review history.
Innovation Solution
A method that updates reliability information for each worker based on their task results, using a process that iteratively refines the comprehensive task result by minimizing error values, allowing for accurate weighting of reviewer abilities and adapting to current task types, even for workers with no prior review history.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a majority vote method is used to determine task results with equal weighting of all reviewers, then the determination process is simple and fast, but the reliability of task result determination is lowered due to not accounting for varying review abilities
Solution Approach 1:
The patent applies local quality by assigning different weights to different reviewers based on their individual review abilities. Instead of treating all reviewers equally (uniform quality), the system evaluates each reviewer's performance on specific task types and assigns localized weightings that reflect their actual competence in those areas. This allows the system to leverage each reviewer's strengths while mitigating weaknesses.
Solution Approach 2:
The patent changes the parameter of reviewer weighting from a fixed equal value to a dynamic value that varies based on review ability assessments. The system continuously updates reviewer weights by analyzing past performance data, task type compatibility, and review quality metrics. This parameter transformation enables the system to adapt to varying reviewer competencies while maintaining computational efficiency.
2Reliability
If review ability is calculated based on past review results, then reviewer weights can be determined, but appropriate weights cannot be applied when the type of work reviewed in the past is different from the type of work reviewed currently
Solution Approach 1:
The patent segments the review ability assessment by task type, creating separate weight calculations for different categories of work. Instead of a single global reviewer rating, the system divides review abilities into task-specific components, allowing reviewers to be evaluated and weighted differently based on their expertise in particular domains. This segmentation enables accurate weighting even when reviewers transition between different task types.
Solution Approach 2:
The patent implements dynamic weight adjustment that adapts to current task requirements. Reviewer weights are not fixed but are continuously updated based on the specific task at hand, the reviewer's historical performance on similar tasks, and real-time feedback. This dynamic approach allows the system to flexibly allocate reviewer weights according to current needs rather than relying on static historical averages.
3Ease of manufacture
If conventional crowdsourcing methods are used without reliability information, then the system is easier to implement, but the quality of labeled data cannot be ensured
Solution Approach 1:
The patent implements self-service by having the system automatically evaluate and update reviewer reliability weights without requiring external intervention. The system continuously monitors review outcomes, compares results against ground truth when available, and autonomously adjusts reviewer weights. This self-service mechanism maintains high data quality while keeping the implementation straightforward, as the complexity is encapsulated within automated algorithms rather than manual processes.
Solution Approach 2:
The patent incorporates feedback loops where review results are continuously evaluated and used to update reviewer reliability assessments. The system collects feedback from multiple sources including task outcomes, reviewer consistency, and agreement with expert validations. This feedback mechanism systematically improves data quality by identifying and weighting high-performing reviewers while gradually eliminating the influence of lower-quality contributions.
Data Source
AI summary
The present invention relates to a method, a system, and a computer-readable medium for deriving task results by reflecting reliability information of a worker processing a work collected through crowdsourcing, and more particularly, to a method, a system, and a computer-readable medium for deriving task results by reflecting reliability information of a worker processing a work collected through crowdsourcing, in which upon a worker processes a work through crowdsourcing, reliability information (working or reviewing ability) for each worker is updated, and a task result for the corresponding work is derived based on the updated reliability information, so as to effectively infer the task result for the work.