Spatial Attention Pretraining for Visually Rich Document Layout Reading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Large Language Models (LLMs) lack the ability to perform spatial-aware reading for visual documents, which is crucial for understanding diagrams, charts, and extracting information from complex visual content, as they do not integrate spatial features effectively.
Innovation Solution
Implementing a layout-aware generative module with disentangled spatial attention architecture and autoregressive block-infilling pretraining objective to adapt text semantics to spatial-aware multi-modal semantics, ensuring continuous learning and addressing disconnected predictions in visual documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing Large Language Models are used for visual document understanding, then the models can process text input, but they fail to perform spatial-aware reading and understand layout information
Solution Approach 1:
The patent extends the model's capability from one-dimensional text processing to two-dimensional spatial-aware processing by introducing spatial position embeddings and a spatial attention module. This allows the model to simultaneously process both text content and its spatial layout information, enabling tasks like visual question answering that require understanding of document structure and element positions.
2Ease of operation
If disconnected fragments are processed without spatial constraints, then the model can handle isolated text segments, but it cannot make coherent predictions across the document layout
Solution Approach 1:
The patent merges the text processing stream with the spatial layout stream by integrating spatial position embeddings into the text embeddings and introducing a spatial attention module. This unified representation allows the model to process isolated fragments while maintaining awareness of their spatial relationships, ensuring coherent predictions across the entire document layout.
3Productivity
If standard next-token prediction is used, then the model can generate text sequentially, but it cannot perform block-level infilling for document reconstruction
Solution Approach 1:
The patent segments the document into meaningful blocks (such as paragraphs, sections, or visual elements) and performs infilling at the block level rather than individual token level. This segmentation approach maintains the efficiency of sequential generation while significantly improving reconstruction accuracy for document-level tasks, as the model can predict entire semantic units rather than isolated words.
Data Source
AI summary
Various methods and processes, apparatuses/systems, and media for performing spatial-aware reading for visual documents are disclosed. A processor implements a language model; modifies existing parameters and architecture of the language model to incorporate new parameters in its architecture by integrating a disentangled spatial attention process to the language model; pretrains the modified model by performing an autoregressive block infilling process on a plurality of document pages thereby training the new parameters and further adjusting the existing parameters; instruction-tunes the pretrained model on data derived from a plurality of visually rich document understanding datasets to teach the pretrained model to follow document-oriented instructions or answer questions about documents by leveraging their content and their spatial layout and outputting a trained model; and performs spatial-aware reading for visual documents by utilizing the trained model.