Continuous Active Machine Learning for Document Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document review systems face inefficiencies due to human reviewers waiting for machine-classified documents, as the time taken by machines to reclassify documents after model updates is significant, leading to prolonged wait times for human reviewers and reduced productivity.
Innovation Solution
Implementing a system with a primary queue and multiple secondary queues, where documents are continuously classified and updated using continuous active machine learning, allowing human reviewers to work on the most relevant documents while the models are being retrained, enabling asynchronous processing of document selection, review, and model retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine classification and model retraining are performed sequentially, then model accuracy is improved, but human reviewer wait time increases
Solution Approach 1:
The system performs preliminary classification of documents into multiple queues before retraining, maintaining ready-to-review documents in advance. When retraining occurs, already-classified documents remain available, preventing reviewer idle time.
Solution Approach 2:
The system maintains continuous document classification across multiple queues, ensuring that while one queue is being retrained, other queues continue to provide reviewable documents, eliminating interruptions in the review workflow.
2Measurement precision
If multiple document queues are maintained for different models, then document classification accuracy is improved, but system complexity increases
Solution Approach 1:
The document set is segmented into multiple queues, each managed by a separate model. This segmentation allows independent optimization of each queue without affecting others, improving overall classification accuracy while managing complexity through modular organization.
Solution Approach 2:
The system uses a universal document structure that can be processed by multiple different models and queues. This multi-functionality allows the same document framework to serve various classification purposes, reducing overall system complexity despite having multiple queues.
3Adaptability or versatility
If continuous retraining is performed, then model relevance is improved, but processing time is increased
Solution Approach 1:
The system implements periodic retraining at scheduled intervals rather than continuous retraining. This approach maintains model relevance through regular updates while avoiding the excessive processing time that would result from continuous retraining operations.
Solution Approach 2:
The system prepares documents for retraining in advance by maintaining them in structured queues with metadata ready for model processing. This preliminary organization reduces the actual retraining processing time while maintaining model relevance.
Data Source
AI summary
Systems and methods for predictive document coding using continuous active machine learning are described herein. A method uses both a primary queue and a plurality of secondary queues, where each secondary queue is associated with a model for category of documents. The method also repeatedly classifies new batches selected from a large set of documents that have not been reviewed. The classification uses the plurality of models and updates the secondary queues from the best documents in the most recently classified batch. While the method transparently cycles through batches, the most relevant documents are provided to one or more human reviewers from secondary queues via a primary queue. The reviewer confirms relevance or non-relevance in each of the documents for each of the categories. Periodically all the models are retrained using the set of reviewed documents after a selectable number of documents have been reviewed since the most recent retraining.