Virtual Adversarial Training for Document Information Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information extraction models for business documents require significant manual annotation and resource-intensive training, often leading to error-prone results, especially when dealing with domain-specific documents, and lack efficient methods to optimize pretraining and fine-tuning processes.
Innovation Solution
A method involving pretraining on unlabeled documents using dynamic window-based informative word masking followed by fine-tuning with virtual adversarial training (VAT) to generate a domain-specific neural network model for accurate document information extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual annotation and traditional training methods are used for information extraction models, then the models can be trained on domain-specific documents, but the training process becomes resource-intensive and error-prone
Solution Approach 1:
The patent applies pretraining on unlabeled domain-specific documents before fine-tuning on labeled data. This preliminary action allows the model to learn domain-specific language patterns and structures without requiring manual annotations, reducing both computational resources needed during fine-tuning and improving reliability by adapting to domain-specific terminology and document formats
Solution Approach 2:
The model performs self-supervised learning by masking word tokens and predicting them from context during pretraining. This self-service mechanism eliminates the need for manual annotation of unlabeled documents, reducing computational resources required for data preparation while improving extraction accuracy through domain-specific pretraining
2Reliability
If traditional pretraining and fine-tuning processes are used, then models can be trained on domain-specific data, but the training time becomes excessive
Solution Approach 1:
The patent performs pretraining on unlabeled domain-specific documents before fine-tuning, which serves as a preliminary action that accelerates convergence during fine-tuning. This approach maintains model performance by learning domain-specific patterns upfront, while reducing total training time by avoiding the need for extensive fine-tuning on labeled data
Solution Approach 2:
The patent extracts and utilizes unlabeled domain-specific documents for pretraining, separating this resource-intensive task from the time-critical fine-tuning phase. By taking out the pretraining step and performing it independently on abundant unlabeled data, the model achieves better performance with reduced overall training time
3Ease of manufacture
If uniform masking rates are applied during pretraining, then the pretraining process is simpler, but informative word tokens are not adequately prioritized
Solution Approach 1:
The patent applies different masking rates to different word tokens based on their informativeness, rather than using a uniform rate. This local quality approach prioritizes informative word tokens (such as domain-specific terminology and key entities) with lower masking rates, ensuring they are adequately represented during pretraining, while maintaining simplicity through automated informativeness calculation
4Reliability
If virtual adversarial training is applied to labeled documents, then the model becomes more robust to perturbations, but the fine-tuning process becomes more complex
Solution Approach 1:
The patent introduces virtual adversarial perturbations as an intermediary during fine-tuning to improve model robustness. These perturbations are added to the input documents during training, creating a mediator that helps the model learn to handle variations and noise, thereby improving robustness while maintaining manageable complexity through automated perturbation generation
Data Source
AI summary
The present disclosure relates to computer-implemented methods, software, and systems for extracting information from documents based on training techniques to generate a document foundation model that is used to initialize a document information extraction model that is fine-tuned to business document specifics. A document information extraction model is initialized based on weights provided from a first pretrained model. Fine-tuning of the document information extraction model is performed based on labeled business documents as second training data. The labeled business documents are labeled and evaluated according to a virtual adversarial training (VAT). Based on the performed fine-tuning, a classifier for classification of information extraction is generated.