Crowdsourced Document Classification with Iterative Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current automatic document classification methods are often inaccurate and prone to manipulation, lacking the reliability of human judgment, and are not practical for large volumes of content, especially in the context of increasing product-related documents and social media posts.
Innovation Solution
A system that combines automated classification with human judgment through a crowdsourcing approach, where high-confidence classifications are validated by analysts and unclear or invalid classifications are refined, with iterative training of classification models using feedback from both automated and human validation to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated classification methods are used, then processing speed and productivity are improved, but classification accuracy and reliability deteriorate
Solution Approach 1:
The classification process is segmented into multiple stages: automated classification for high-volume processing, followed by human analyst review for uncertain cases. This segmentation allows the system to leverage the speed of automation while maintaining accuracy through selective human intervention.
Solution Approach 2:
Human analysts provide feedback on automated classification results, particularly for uncertain or low-confidence classifications. This feedback loop continuously improves the automated system's accuracy while maintaining high processing throughput for clear-cut cases.
2Reliability
If human classification is used, then classification accuracy and reliability are improved, but processing capacity and productivity deteriorate
Solution Approach 1:
Instead of having humans classify all documents, the system applies partial human action only to uncertain or low-confidence automated classifications. This selective approach maintains high accuracy where needed while preserving overall processing capacity.
Solution Approach 2:
The automated classification system serves itself by handling the majority of classifications independently, only escalating uncertain cases to human analysts. This self-service capability maximizes processing capacity while maintaining reliability through targeted human review.
3Productivity
If automated classification is used, then processing efficiency is improved, but susceptibility to manipulation and spam increases
Solution Approach 1:
Human analysts serve as an intermediary layer between automated classification and final results, particularly for suspicious or manipulated content. This intermediary detection capability maintains processing efficiency while providing robust defense against spam and manipulation attempts.
Data Source
AI summary
Systems and methods are disclosed herein for classifying records, such as product records, using a machine learning algorithm. After training a classification model according to a machine learning algorithm using an initial training set, records are classified and high confidence classifications identified. Remaining classifications are submitted to a crowdsourcing forum that validates or invalidates the classifications or marks them as to unclear to evaluate. Invalidated classifications are automatically analyzed to identify one or both of classification values and categories having a high proportion of invalidated classifications. Requests are transmitted to analysts to generate training data that is added to the training set. The process of classifying records and obtaining crowdsourced validation thereof may then repeat. High confidence classifications may be identified using an accuracy model trained to relate an accuracy percentage to a confidence score output by the classification model.