Worker Group Identification for Accurate Crowdsourcing

The patent segments the worker population into different groups based on their characteristics and performance. By dividing workers into segments (e.g., high-performing vs. low-performing workers), the system can selectively combine judgments from specific segments rather than all workers, thereby maintaining accuracy while still achieving quantity through parallel processing across multiple worker groups.

The patent applies local quality by treating different worker groups differently based on their demonstrated capabilities. High-performing workers receive more weight or their judgments are prioritized in the aggregation process, while lower-performing workers contribute less. This differential treatment based on local worker quality maintains overall label accuracy even as the quantity of labels increases through crowdsourcing.

The patent performs preliminary action by evaluating and characterizing workers before they participate in the main labeling task. Through initial assessments, performance metrics, and characteristic detection, the system pre-identifies high-performing workers who will be relied upon for critical labeling decisions. This preliminary sorting ensures that when productivity is increased by engaging many workers, the reliability is maintained through selective use of pre-identified competent workers.

The patent changes parameters by dynamically adjusting the weight, influence, or selection criteria for different workers based on their measured characteristics and performance history. Workers with higher accuracy metrics, appropriate expertise attributes, or better performance on test items receive higher parameter values in the aggregation function, allowing the system to maintain reliability while scaling productivity by incorporating more workers with varying parameter weights.