Automated Document Review Model Training via Peer Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Human reviewers in electronic document review often exhibit inconsistency and inefficiency, leading to inaccurate and time-consuming processes, especially when machine learning models rely on flawed manually coded training data.
Innovation Solution
A system that calculates reviewer accuracy and speed by peer review, divides documents into control, training, and review sets, trains a predictive model using these documents, and assigns documents based on effective speed scores to improve the automated review process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If machine learning methods are used to train a predictive model to automatically review electronic documents, then the overall consistency of reviews increases and costs decrease, but the accuracy of the model is dependent on the accuracy of the training data which may be manually coded and flawed
Solution Approach 1:
The system performs preliminary actions by implementing a multi-stage review process where documents are first reviewed by multiple human reviewers, then used as training data for the predictive model. Quality control documents are reviewed again after model training to verify accuracy, ensuring the model is trained on high-quality data before automated review begins
Solution Approach 2:
The system implements feedback mechanisms where the predictive model's outputs are continuously evaluated against ground truth from manually reviewed documents. The model accuracy is measured and used to improve subsequent training iterations, creating a closed-loop system that enhances model reliability over time
2Productivity
If documents are allocated to multiple reviewers for faster review, then the review speed increases, but reviewers may not be consistent with each other and each reviewer may have a different level of accuracy or speed
Solution Approach 1:
The system changes parameters by calculating effective speed scores that combine both accuracy and speed metrics for each reviewer. Documents are dynamically allocated based on these parameters, matching documents with appropriate reviewers according to their demonstrated performance characteristics rather than using static assignment methods
Solution Approach 2:
The review process is segmented into distinct phases: initial review by multiple reviewers, quality control review of a subset, and final automated review by the predictive model. This segmentation allows different consistency requirements to be applied at different stages, with early stages benefiting from multiple reviewers and later stages relying on the trained model
3Measurement precision
If manual coding is used to review electronic documents, then the review process can be performed, but it takes valuable time and resources and may not be cost-effective for large volumes of electronic documents
Solution Approach 1:
The system introduces an intermediary predictive model that learns from manually reviewed documents and then performs automated reviews. This intermediary translates the expertise of human reviewers into an automated system, reducing the need for extensive manual coding while maintaining accuracy through the model's learning capability
Solution Approach 2:
The system applies partial manual action by having human reviewers code only a subset of documents (control documents and quality control documents) rather than all documents. This partial manual coding is sufficient to train the predictive model, which then handles the remaining documents automatically, reducing overall time and resource requirements
Data Source
AI summary
A computer-implemented method for automated document review and quality control may include (1) dividing a set of documents to be reviewed for relevancy into sets of control documents, training documents, quality-control documents, and review documents, (2) calculating, based on a set of reviews performed by a group of reviewers on the set of quality-control documents, an effective speed score for each reviewer in the group of reviewers, (3) assigning, based on the effective speed score, the set of control documents and the set of training documents to the group of reviewers, (4) training a predictive model using a set of training reviews performed by the group of reviewers on both the set of training documents and the set of control documents, and (5) using the predictive model to evaluate the set of review documents. Various other methods, systems, and computer-readable media are also disclosed.