Document Information Extraction Model Pretraining with Informative Word Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information extraction methods face challenges in efficiently processing large volumes of business documents, particularly due to the labor-intensive and error-prone process of manual annotation, and the need for domain-specific annotated datasets.
Innovation Solution
The method involves pretraining a document information extraction model using unlabeled documents with dynamic window adjustments and informative word masking, followed by fine-tuning with labeled business documents using virtual adversarial training (VAT) techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual annotation is used for training data preparation, then model training can be performed, but the process becomes labor-intensive and error-prone
Solution Approach 1:
The system performs self-supervised pretraining by automatically generating training signals from unlabeled documents through informative word masking, eliminating the need for manual annotation. The model learns to predict masked words based on contextual information, with the masking process itself providing the training supervision signal.
Solution Approach 2:
The patent applies pretraining on unlabeled documents before fine-tuning on labeled data. This preliminary action allows the model to learn domain-specific patterns and representations from abundant unlabeled business documents, reducing the burden of manual annotation required for subsequent fine-tuning.
2Reliability
If domain-specific annotated datasets are created, then model performance improves, but the data preparation process becomes more complex
Solution Approach 1:
The system automatically identifies informative words and generates masking patterns without human intervention. The informative word identification process uses statistical measures (such as pointwise mutual information) to automatically determine which words should be masked, eliminating the need for manual dataset curation.
Solution Approach 2:
The patent performs pretraining on domain-specific unlabeled documents before fine-tuning. This preliminary domain adaptation allows the model to learn business document characteristics without requiring manually annotated domain-specific training data, simplifying the data preparation process.
3Measurement precision
If extensive manual annotation is performed, then training data quality improves, but processing efficiency decreases
Solution Approach 1:
The model generates its own training signals through the informative word masking process. By automatically identifying which words are informative and should be masked, the system creates high-quality training data without manual annotation, maintaining training data quality while dramatically improving processing efficiency.
Solution Approach 2:
The patent changes the parameter of masking rate from uniform to variable based on word informativeness. This parameter change allows the model to focus learning on the most informative words, improving training data quality while reducing the overall amount of annotation required.
Data Source
AI summary
The present disclosure relates to computer-implemented methods, software, and systems for extracting information from business documents based on training techniques to generate a document foundation model by pretraining. First training data based on a plurality of unlabeled documents is obtained for use in training a first model for document information extraction. The first model is pretrained according to a dynamic window adjustable to a word token count for each document of the plurality of unlabeled documents. The pretraining comprises evaluating word tokens in each of the plurality of unlabeled documents where masking is applied according to individual masking rates determined for the word tokens. The individual masking rates are indicative of respective informative relevance of the word tokens. The pretrained first model is provided for initializing a second document information extraction model to be trained based on labeled documents as second training data.