Crowdsourcing Task Assignment with Per-Worker Answer Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Requesters face challenges in determining the appropriate payment for workers and the number of workers needed to achieve desired accuracy when outsourcing tasks to crowdsourcing platforms, and they struggle to assess the accuracy of answers received from multiple human workers.
Innovation Solution
A method and system that analyze answers on a per-worker and per-question basis to determine adjustments, allowing for optimal task submission parameters and post-analysis to ensure accuracy and quality of results, including withholding payment or discarding answers from inconsistent workers and adjusting for poorly crafted questions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If multiple workers are employed to complete tasks on a crowdsourcing platform, then the quantity of answers increases, but the accuracy and quality of the final answer becomes difficult to ascertain
Solution Approach 1:
The system implements a feedback mechanism where the analysis module evaluates worker performance based on answer quality and consistency. This feedback is used to dynamically adjust worker selection, payment amounts, and task assignment strategies, thereby improving answer accuracy while managing the number of workers effectively
Solution Approach 2:
The system changes parameters such as payment amounts and number of workers required based on task complexity and desired accuracy levels. By dynamically adjusting these parameters, the system optimizes the balance between obtaining sufficient answers and ensuring their accuracy
2Reliability
If the number of workers is increased to improve answer accuracy, then the reliability of results improves, but the cost of task completion increases
Solution Approach 1:
The system applies partial action by determining the minimum number of workers needed to achieve the desired accuracy level for each task. Rather than uniformly employing a fixed number of workers for all tasks, the system calculates and applies only the necessary workforce, avoiding excessive spending while maintaining reliability
Solution Approach 2:
The system dynamically changes the number of workers and payment parameters based on task characteristics and required accuracy. This allows optimization of the relationship between reliability and cost by adjusting parameters to match specific task requirements
3Productivity
If payment amounts are increased to attract more workers, then the productivity of task completion improves, but the cost per task increases
Solution Approach 1:
The system makes payment amounts dynamic rather than static. Payment parameters are adjusted based on task complexity, worker performance history, and market conditions, allowing the system to optimize productivity while controlling costs by paying appropriately for each specific task
4Measurement precision
If answers are analyzed on a per-worker basis to improve quality control, then the measurement precision of worker performance improves, but the device complexity of the analysis system increases
Solution Approach 1:
The analysis system is segmented into modular components that handle different aspects of worker evaluation independently. This segmentation allows for precise per-worker analysis while managing system complexity through organized, separate processing modules for different evaluation criteria
Data Source
AI summary
A method and system for applying a task to a crowdsourcing platform for processing are disclosed. For example, a method forwards the task having a question to the crowdsourcing platform, and receives a plurality of answers to the question, wherein the plurality of answers is provided by a plurality of workers associated with the crowdsourcing platform. The method analyzes the plurality of answers on a per-worker basis to determine an adjustment, and applies the adjustment to the plurality of answers.