Active Learning Workflow for Continuous Document Model Updating
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning models for document analysis require time-consuming and oracle-dependent sequential processes for annotation, which are costly and inefficient, especially when dealing with confidential documents.
Innovation Solution
An active learning and automatic analysis system that operates in parallel learning and production modes, continuously improving the machine learning model by automatically sampling, annotating, and validating documents, using a pre-trained portion for faster updates and non-expert implementation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional machine learning approaches are used with manual annotation of training documents, then model training can be performed, but the process is time-consuming and costly due to requiring extensive manual annotation of training documents
Solution Approach 1:
The system uses the production model to automatically analyze unannotated documents and generate descriptors, allowing the system to serve itself by creating training data without external human annotation for every document. The active learning component selectively identifies which documents need human annotation, reducing the overall annotation burden.
Solution Approach 2:
The system performs preliminary automatic analysis of unannotated documents using the production model before they are selected for human annotation. This preliminary action creates initial descriptors that guide the active learning selection process, preparing documents in advance for efficient training data creation.
2Quantity of substance
If all unannotated documents are annotated and used for training, then training data quantity increases, but the cost and time consumption increase significantly
Solution Approach 1:
The active learning component changes the parameter of document selection by using descriptor-based filtering and sampling to identify only the most informative unannotated documents for human annotation. This selective approach changes from annotating all documents to annotating only those that maximally improve model performance.
Solution Approach 2:
The system performs partial annotation by selecting only a subset of unannotated documents for human annotation based on active learning criteria. This partial action is sufficient to maintain and improve model performance without the excessive cost of annotating all available documents.
3Reliability
If the production model is frequently updated with new training data, then model performance improves, but the stability of the production system may be affected
Solution Approach 1:
The system implements dynamic model updating where the production model is replaced with candidate models based on validation performance. This dynamic approach allows the system to adapt and improve performance while maintaining stability through systematic validation and controlled replacement rather than frequent arbitrary updates.
Solution Approach 2:
The system uses validation sets to provide feedback on candidate model performance before deployment to production. This feedback mechanism ensures that only models that improve performance are deployed, maintaining system stability while enabling continuous performance improvement through controlled updates.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
An active learning and machine analysis system runs a training mode and a production mode in parallel. In production mode, it responds to requests for automatic document analysis using a machine learning model trained on annotated documents. In training mode, it receives and stores unannotated documents, updates a descriptor with predictive information from the automatic analysis of the unannotated documents, samples the stored unannotated documents whose descriptor has been updated, and determines a schedule for annotation of the sampled unannotated documents by an oracle. It allocates the annotated documents between those for training and those for validation. It trains at least one candidate machine learning model with a random structure, which, if it performs better in validation, replaces the model used in production mode.A new training session is then performed by updating the descriptor according to the replacement model.