Graph Networks for Cross-Domain Visually Rich Document Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training a machine learning model to perform key-value extraction from visually rich documents requires large amounts of labeled data, and existing models struggle to generalize across different domains due to insufficient variation in training data.
Innovation Solution
A method involving the use of pre-trained language and visual models, combined with positional embeddings, to extract and classify key-value pairs from visually rich documents, utilizing a graph neural network to integrate textual, visual, and positional features, allowing for training with minimal labeled data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If supervised machine learning algorithms are trained on general domain data, then the model can learn basic patterns, but the model struggles to classify data from different target domains due to insufficient domain adaptation
Solution Approach 1:
The patent applies preliminary action by pre-training language models on general domain data before fine-tuning them for specific target domains. This pre-training establishes foundational patterns that can be subsequently adapted to different domains with minimal additional training data, thereby improving domain adaptability while maintaining classification accuracy
Solution Approach 2:
The patent utilizes parameter changes by adjusting model weights and parameters during domain-specific fine-tuning. The system modifies the pre-trained model's parameters to adapt to target domain characteristics, enabling the model to generalize across different domains while maintaining reliable classification performance
2Reliability
If large amounts of labeled data are used for training, then the model can learn sufficient variations to generalize, but the training process requires considerable time and computational resources
Solution Approach 1:
The patent applies preliminary action by using pre-trained language models that have already learned general patterns from large corpora. This eliminates the need to train from scratch on extensive labeled data, achieving model generalization with significantly reduced training time and computational resources
Solution Approach 2:
The patent utilizes copying by leveraging knowledge transferred from pre-trained models through fine-tuning. Instead of creating new training data and models from scratch, the system copies and adapts existing pre-trained model weights to specific domains, achieving generalization efficiency with minimal labeled data
3Loss of information
If traditional machine learning models are used for key-value extraction, then the models can process text data, but they fail to effectively utilize visual features and positional information from visually rich documents
Solution Approach 1:
The patent applies merging by integrating language models with visual processing capabilities and positional encoding mechanisms. This combination allows the model to simultaneously process textual, visual, and spatial information from visually rich documents, preventing information loss while maintaining manageable architectural complexity through modular design
Data Source
AI summary
In some implementations, techniques described herein may include identifying text in a visually rich document and determining a sequence for the identified text. The techniques may include selecting a language model based at least in part on the identified text and the determined sequence. Moreover, the techniques may include assigning each word of the identified text to a respective token to generate textual features corresponding to the identified text. The techniques may include extracting visual features corresponding to the identified text. The techniques may include determining positional features for each word of the identified text. The techniques may include generating a graph representing the visually rich document, each node in the graph representing each of the visual features, textual features, and positional features of a respective word of the identified text. The techniques may include training a classifier on the graph to classify each respective word of the identified text.