Crowdsourced Training Data Allocation Using Assessor Class Scores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing crowdsourcing platforms face challenges in ensuring the correctness of human-assessor outputs for training machine learning algorithms, leading to inefficient use of resources and suboptimal performance due to the allocation of tasks to unsuitable assessors, particularly when encountering new task types.
Innovation Solution
A method and system that analyze past performance data of assessors across different task types to rank and assign tasks based on class scores, ensuring that assessors with high proficiency in similar tasks are allocated new tasks, thereby improving the accuracy of training data generation for machine learning algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If tasks are allocated to human assessors without considering their suitability for specific task types, then the platform can operate with simpler task allocation mechanisms, but the correctness of training data and performance of machine learning algorithms deteriorates
Solution Approach 1:
The system performs preliminary analysis of assessor performance data across different task types before task allocation. Class scores are pre-calculated based on historical performance, allowing the platform to make informed allocation decisions that improve training data correctness without requiring complex real-time evaluation mechanisms
Solution Approach 2:
The system transforms raw performance data into standardized class scores that quantify assessor suitability for different task types. This parameter transformation enables simple comparison and ranking of assessors, resolving the contradiction by creating a straightforward scoring mechanism that improves reliability while maintaining allocation simplicity
2Measurement precision
If assessors are selected based on their past performance in similar task types, then the accuracy of training data generation improves, but the time and computational resources required for assessor evaluation and selection increase
Solution Approach 1:
Performance analyses and class score calculations are performed in advance before task allocation occurs. By pre-processing assessor data and establishing performance metrics beforehand, the system eliminates time-consuming evaluations at task assignment moment, thus improving measurement precision without proportionally increasing time loss
Solution Approach 2:
The system uses historical performance patterns as proxies for future task performance. By analyzing past performance in similar task types and using these patterns to predict future accuracy, the system achieves high measurement precision through efficient data reusing rather than extensive new evaluations
3Reliability
If the platform allocates additional monetary resources to compensate more assessors for ensuring correct outputs, then the correctness of training data improves, but the operational cost increases
Solution Approach 1:
The system applies differentiated compensation strategies based on local assessor characteristics and task requirements. By identifying assessors with high class scores for specific task types and selectively engaging them, the platform achieves high output correctness while minimizing monetary resources by not over-compensating all assessors uniformly
Solution Approach 2:
The system uses feedback from past task performance to continuously refine assessor class scores and allocation decisions. This feedback mechanism ensures that monetary resources are directed toward assessors who demonstrably produce correct outputs, improving reliability while optimizing resource allocation efficiency
4Adaptability or versatility
If the system analyzes past performance data across multiple task types to generate class scores, then the adaptability to new task types improves, but the complexity of data processing and analysis increases
Solution Approach 1:
The system creates universal class scores that capture assessor performance across multiple task types. These multi-functional scores can be applied to various new task types without requiring task-specific evaluation mechanisms, thus improving adaptability while managing data processing complexity through score generalization
Solution Approach 2:
The system merges performance data from multiple task types into unified class scores for each assessor. By combining diverse performance indicators into integrated scores, the system achieves broad adaptability to new task types while reducing the complexity of analyzing multiple separate metrics
Data Source
AI summary
Non-limiting embodiments of the present technology are directed to a method and system for generating a training dataset. The method comprises: accessing data associated with a plurality of assessors executing digital tasks of a first type and digital tasks of a second type; generating, a first ranked list of assessors and a second ranked list of assessors based on their past performance; for a given one of the plurality of assessors: generating, a class score for the common class of digital tasks; acquiring a request for executing a digital task of a third type; ranking, the plurality of assessors based on respective class scores, the given one from the plurality of assessors being one of top ranked ones from the plurality of assessors; transmitting the digital task of the third type to the given one; generating the training data for the MLA based on a response from the given one.