Self-Attentive Key-Value Extraction for Low-Annotation Documents
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual annotation of documents for neural network-based extraction models is labor-intensive, and adding new fields or supporting new document types requires annotating thousands of documents, making maintenance difficult due to small label set overlaps and evolving label sets.
Innovation Solution
Automated extraction of key-value pairs using a system that maps word tokens to vectors, determines attention scores, normalizes them, and identifies connected tokens, minimizing computational resources and memory requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual annotation is used to train neural network-based extraction models, then the models can be trained with traditional methods, but the process becomes labor-intensive and requires annotating thousands of documents when adding new fields or document types
Solution Approach 1:
The system uses pre-trained language models to automatically annotate documents without requiring human labelers. The model extracts key-value pairs autonomously by processing document text through transformer architectures, eliminating the need for manual annotation while maintaining extraction accuracy
Solution Approach 2:
The system performs preliminary actions by pre-training language models on large corpora before deployment. This pre-training enables the model to understand document structures and extract key-value pairs effectively without requiring extensive manual annotation for each specific document type
2Adaptability or versatility
If thousands of documents are annotated to support new document types or fields, then the models can maintain usability, but the maintenance process becomes difficult and time-consuming
Solution Approach 1:
The system uses a universal transformer-based language model that can process multiple document types and extract various key-value pairs using the same architecture. The model adapts to different document types through its pre-trained language understanding rather than requiring separate annotation schemas for each type
Solution Approach 2:
The system changes parameters by adjusting the input text and extraction targets rather than changing the fundamental model architecture. When new document types or fields are needed, the system processes new text data through the existing transformer model, avoiding the need to re-annotate thousands of documents
3Quantity of substance
If extensive manual annotation is performed, then training data is available, but computational resources and memory requirements increase significantly
Solution Approach 1:
The system uses copying by leveraging pre-trained language models that have already learned from large corpora. Instead of training from scratch on manually annotated data, the system copies knowledge from pre-trained models and applies it to specific extraction tasks, reducing computational resource requirements
Data Source
AI summary
Systems and methods are provided for automated identification of key-value pairs in documents. A document including readable text is received. The document is processed to determine, from the readable text, a plurality of tokens. Pairs of vectors corresponding to the plurality of tokens are determined, each pair of vectors comprising a query vector and a key vector. Attention scores are determined for the plurality of tokens by using the pairs of vectors. The attention scores are normalized to generate normalized attention scores. Connected tokens are identified in the plurality of tokens using the normalized attention scores.