Crowdsourcing Contribution Value Calculation for Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing crowdsourcing techniques face challenges in ensuring high operation accuracy due to varying worker quality and inefficiencies in existing methods, such as majority schemes and verification schemes, which struggle to achieve target accuracy with a smaller number of participants.
Innovation Solution
A method that calculates a contribution value for each participant based on their correct answer probability using a binomial distribution, adding this value to determine the necessary number of participants required to achieve target accuracy, and outputs the most likely correct answer when the sum of contribution values exceeds a predetermined condition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a majority scheme is used to determine task results, then the decision-making process is simple, but the operation accuracy cannot reach the target level and efficiency is low
Solution Approach 1:
The patent transforms the qualitative majority voting process into a quantitative calculation by introducing contribution values based on correct answer probabilities. Each participant's answer is weighted by their calculated contribution value, and the result is determined by summing these weighted values rather than simple majority counting. This parameter change enables achieving target accuracy while maintaining operational simplicity.
2Reliability
If expert verification is used to ensure accuracy, then the operation accuracy is improved, but processing efficiency decreases due to bottlenecks
Solution Approach 1:
The system enables self-service by allowing participants to automatically contribute to task completion based on their calculated contribution values. Instead of requiring expert verification of each answer, the system automatically aggregates weighted answers from multiple participants, with the cumulative contribution values determining task completion. This eliminates the expert verification bottleneck while maintaining accuracy.
3Reliability
If additional majority scheme with two persons is used, then accuracy is improved, but the system requires combination with verification scheme and has small expandability
Solution Approach 1:
The contribution value calculation system serves multiple functions: it can handle tasks with any number of participants, works with different target accuracy levels, and automatically adjusts the number of required participants based on individual contribution values. This universal approach replaces the need for fixed-pair majority schemes and their accompanying verification mechanisms, providing both accuracy and expandability.
4Reliability
If only reliable workers are used to perform tasks, then operation accuracy is maintained, but the system cannot handle situations where less reliable workers are also contained
Solution Approach 1:
The system applies local quality by assigning different contribution values to different participants based on their individual correct answer probabilities. Rather than uniformly excluding less reliable workers, each participant's answer is weighted according to their specific reliability level. This allows the system to incorporate inputs from workers with varying reliability while maintaining overall accuracy through the weighted aggregation mechanism.
Data Source
AI summary
A contribution value necessary for achieving a target accuracy from a correct answer probability of each participant is calculated, a contribution value of the participant is added for an answer in accordance with the calculation, and it is set at a condition for determining completion of a task, that is, determining that a correct answer is obtained and no additional participant is necessary. The contribution value is calculated as the inverse of the number of participants at which the target accuracy is reached with a predetermined correct answer probability. The contribution value of the participant is added to the contribution value for the task in which that participant participates. At the time when the sum of the contribution values for a task exceeds one or when one option is certain to be a correct answer, the result having the largest sum of the contribution values is output.