VQA Document Adapters for Targeted Extraction Across Document Types
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing optical character recognition systems struggle to understand relationships and hierarchies between words in scanned documents, particularly when words are not oriented in a structured manner, such as in handwritten text or stickers, and often provide superfluous or missing data, failing to support diverse document types effectively.
Innovation Solution
Implementing a visual question answering (VQA) transformer model with document type specific adapters that adjust tokens and embeddings based on document type specific encoding parameters, using a multi-modal approach to enhance accuracy and support additional document types without overfitting or requiring multiple model versions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If traditional optical character recognition systems are used, then basic text extraction can be performed, but the systems cannot understand relationships and hierarchies between words in scanned documents
Solution Approach 1:
The patent combines the detection subsystem and recognition subsystem into a unified transformer model that processes document images end-to-end. This merging allows the system to simultaneously extract text and understand document structure, relationships, and hierarchies without requiring separate processing stages, thereby reducing information loss while maintaining manageable complexity.
Solution Approach 2:
The patent introduces document type specific adapters as intermediary components between the base transformer model and the final output. These adapters act as mediators that adapt the general-purpose transformer model to specific document types (e.g., forms, receipts, contracts), enabling the system to understand document-specific structures and relationships without requiring complete system redesign for each document type.
2Adaptability or versatility
If traditional OCR systems are used, then structured text can be recognized, but words that are not oriented in a structured way (handwritten text, stickers) cannot be located
Solution Approach 1:
The patent employs a dynamic detection mechanism within the transformer model that can adapt to various text orientations and formats. Unlike traditional systems that assume fixed structured layouts, the transformer model dynamically adjusts its detection approach based on the actual document content, enabling it to locate handwritten text, stickers, and other unstructured text elements with appropriate precision.
Solution Approach 2:
The patent changes the fundamental parameters of text detection by using a transformer model that processes images as sequences of tokens rather than relying on fixed grid-based or contour-based methods. This parameter change enables the system to detect text in various orientations and formats by learning from training data diverse text layouts, thereby improving adaptability while maintaining location precision through the model's attention mechanisms.
3Productivity
If traditional OCR systems are used, then text extraction can be performed, but superfluous data is provided beyond requested data that later needs to be parsed
Solution Approach 1:
The patent extracts only the specific information requested by the user through natural language queries, rather than extracting all possible text from the document. The transformer model with document type specific adapters understands the query intent and selectively extracts only the relevant data fields, eliminating superfluous data extraction and reducing subsequent parsing overhead while improving processing efficiency.
4Adaptability or versatility
If traditional OCR systems are used, then basic document analysis can be performed, but support for additional document types is difficult to provide
Solution Approach 1:
The patent implements a universal transformer model that can handle multiple document types through document type specific adapters. The base model provides general document analysis capabilities, while the adapters enable specialization for different document types (forms, receipts, contracts, etc.). This universal architecture allows the system to support additional document types by simply adding or configuring appropriate adapters without redesigning the entire system, thereby improving versatility while controlling maintenance complexity.
Data Source
AI summary
Document type specific adapters of a document analysis system are used to provide for additional document types or document specialization when generating answers to user submitted questions targeting information included in a document image provided with the user submitted question. The document analysis system receives a visual question answering (VQA) prompt comprising a document image and a question defining information to be extracted from the document image, generates tokens based on the document image and the question, and adjusts encoding of the tokens using document type specific adapters of a transformer model to extract the information from the document image. A classifier of the document analysis system may determine whether the document image matches a document type supported by the document type specific adapters.