Multimodal Language Model Prompting for Document Artifact Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Language models, such as large language models, hallucinate when processing electronic documents containing images, leading to nonsensical or undesirable outputs, particularly in associating artifacts like bounding boxes and text in electronic forms.
Innovation Solution
An augmented language model approach that combines image processing data from OCR and CV applications with a multimodal prompt to enhance the language model's understanding of electronic document structure, reducing hallucination by providing context and structure to the processing instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a language model processes electronic documents containing images, then the model can extract text and perform analysis, but the model generates hallucinations (nonsensical or wrong output)
Solution Approach 1:
The patent segments the document processing task into distinct stages: image processing to extract artifacts and relationships, then language model processing to generate structured output. This segmentation allows each component to optimize for its specific function, reducing hallucinations while maintaining processing capability.
Solution Approach 2:
The patent applies preliminary image processing actions before language model processing to extract and validate artifacts, relationships, and document structure. This preliminary action provides accurate ground truth to the language model, preventing hallucinations while maintaining efficient processing.
2Measurement precision
If image processing data is combined with language model processing, then artifact extraction accuracy improves, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary processing layer that combines image processing data with language model outputs. This intermediary layer validates artifacts, resolves conflicts, and integrates information from both sources, improving accuracy while managing complexity through a structured intermediate representation.
Solution Approach 2:
The patent creates a universal processing framework that handles multiple document types and processing tasks through a single integrated system. The framework universally applies image processing, relationship extraction, and language model processing across different document formats, improving accuracy without proportionally increasing complexity.
Data Source
AI summary
A method including applying an image processing application to an electronic document to generate image processing data describing artifacts in the electronic document. The method also includes applying a server controller to the image processing data to generate at least one relationship among the artifacts. The method also includes converting the at least one relationship and the artifacts into an object notation language data structure. The method also includes generating a multimodal prompt for a language model by combining the object notation language data structure, a reference to the electronic document, and a prompt template. The prompt template includes instructions for the language model to extract segments from the electronic document. The method also includes applying, according to the multimodal prompt, the language model to the electronic document to extract, from the electronic document, the segments and a segment relationship among the segments. The method also includes returning the segments.