Machine-Learning Task Ranking for Assessor Accuracy and Satisfaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing crowdsourcing platforms face issues with the correctness of human assessor outputs affecting machine learning training and the biased selection of tasks by assessors, leading to dissatisfaction and difficulty in completing unfamiliar tasks.
Innovation Solution
A computer-implemented method using a Machine-Learning Algorithm (MLA) to rank digital tasks for an assessor based on assessor interaction and accurate-completion parameters, optimizing a ranking quality parameter to balance user and requester satisfaction, ensuring correct task completion and alignment with assessor preferences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If human assessors are allowed to freely select tasks based on their preferences, then assessor satisfaction is improved, but task completion correctness deteriorates due to biased selection and lack of interest in new task types
Solution Approach 1:
The system dynamically changes the parameter of task presentation order based on assessor profile parameters (task completion history, skill levels, performance metrics) rather than using a fixed or random order. This allows the task selection process to adapt to each assessor's capabilities and preferences, maintaining satisfaction while ensuring appropriate task distribution including new task types.
Solution Approach 2:
The system uses feedback from assessor performance data (completion accuracy, speed, quality ratings) to continuously optimize the task ranking algorithm. This feedback loop enables the system to learn which assessors excel at which task types and adjust task presentation accordingly, improving both assessor satisfaction and completion correctness over time.
2Adaptability or versatility
If assessors are assigned unfamiliar task types, then requester satisfaction is improved through diverse task completion, but assessor satisfaction deteriorates due to lack of familiarity and interest
Solution Approach 1:
The system applies local quality by customizing the task presentation for each individual assessor based on their specific profile characteristics (skill level, experience, performance history). Rather than a uniform approach, each assessor receives a personalized task ranking that optimally balances familiar and unfamiliar tasks according to their individual capabilities and preferences.
Solution Approach 2:
The task ranking is dynamic and adapts as assessors gain experience with new task types. As assessors complete unfamiliar tasks successfully, the system learns from this feedback and gradually increases the proportion of similar new tasks in their queue, creating a dynamic progression that maintains engagement while expanding skill sets.
3Device complexity
If traditional task allocation methods are used, then system complexity is reduced, but training data quality deteriorates due to erroneous labels from biased assessor selection
Solution Approach 1:
The machine learning-based task ranking algorithm serves as an intermediary between task requesters and assessors. This intermediary layer processes multiple parameters (assessor skills, task requirements, performance metrics) to generate optimized task assignments, improving training data quality without requiring direct complex coordination between all parties.
Data Source
AI summary
A method and system for generating a list of digital tasks to a given assessor, the method comprising: receiving, a request for the list of digital tasks from the given assessor; retrieving a plurality of digital tasks available; determining a respective assessor interaction parameter; obtaining a respective accurate-completion parameter; ranking, the plurality of digital tasks to generate a ranked plurality of digital tasks, the ranking being executed by optimizing a ranking quality parameter, the ranking quality parameter being determined based on a combination of: a user-platform satisfaction parameter; a requester-platform satisfaction parameter; selecting, from the ranked plurality of digital tasks, a top N-number of digital tasks for inclusion thereof in the list of digital tasks.