Document Encoder Pretraining With Multimodal Cross-Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional pretraining methods for document understanding fail to adequately account for the hierarchical structure and multimodal content of physical documents, including visual features and spatial layout, leading to inefficiencies and limitations in understanding complex documents.
Innovation Solution
A unified pretraining framework that integrates multimodal self-attention and convolution mechanisms to hierarchically encode local features, using a gated cross-attention transformer to differentiate between textual and visual regions, and employs pretraining tasks like masked sentence modeling, visual contrastive learning, and vision-language alignment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional sequence-to-sequence learning architectures are used to segment documents into word sequences, then the model can process text content, but it fails to adequately account for the hierarchical structure of documents (words form sentences, sentences form semantic regions, and semantic regions form documents)
Solution Approach 1:
The patent segments the document processing task into multiple hierarchical levels: character level, word level, sentence level, and document level. Each level has its own encoding and processing mechanisms, allowing the model to capture structural information at each granularity while maintaining overall document understanding capability.
Solution Approach 2:
The patent introduces a hierarchical dimension to the traditional sequence-to-sequence architecture by adding multiple levels of encoding (character embeddings, word embeddings, sentence embeddings) and multiple attention mechanisms operating at different levels. This transforms the flat sequence processing into a multi-dimensional hierarchical structure that captures both local and global document properties.
2Loss of information
If conventional pretraining methods are used that only receive words as input, then the model can process textual content, but it fails to capture semantic information encoded in visual features such as tables, font sizes, styles, and figures
Solution Approach 1:
The patent merges multiple modalities (visual features including tables, fonts, styles, figures, and textual content) into a unified processing framework. The model integrates these different types of information through shared encoding layers and joint attention mechanisms, allowing visual and textual information to mutually enhance document understanding.
Solution Approach 2:
The patent creates a universal encoder that can process multiple types of input modalities through the same architectural components. The encoding layers and attention mechanisms are designed to handle diverse input types (images, text, tables) uniformly, making the model multi-functional while maintaining architectural efficiency.
3Productivity
If encoder-based document pretraining models are used, then the model can understand text-rich documents, but it suffers from input length constraints as the input length of multi-page documents scales
Solution Approach 1:
The patent segments long documents into smaller hierarchical units (characters, words, sentences, semantic regions) that can be processed independently at lower levels before being aggregated. This segmentation allows the model to handle arbitrarily long inputs by processing them in manageable chunks while preserving global context through hierarchical attention mechanisms.
Solution Approach 2:
The patent addresses the input length problem by introducing hierarchical processing dimensions. Instead of processing the entire document as a single long sequence, the model processes information at multiple scales (local character/word level, intermediate sentence/region level, and global document level), effectively managing long-range dependencies without being constrained by sequence length limitations.
4Measurement precision
If conventional methods encode spatial information via 2D position encoding with single-modality self-attention, then the model can capture positional relationships, but it fails to adequately model spatial relationships in semi-structured documents where words are more related to their local surroundings
Solution Approach 1:
The patent applies local quality by implementing attention mechanisms that adaptively focus on different spatial contexts. The model uses multiple attention heads that can selectively attend to local surrounding words versus global document context, allowing the representation to have different receptive fields depending on the specific processing needs of each position in the document.
Solution Approach 2:
The patent introduces dynamic spatial modeling through attention mechanisms that adaptively adjust their receptive fields. The attention weights are dynamically computed based on the input content, allowing the model to flexibly model spatial relationships - focusing on local surroundings when appropriate and expanding to global context when needed, rather than using fixed positional encodings.
Data Source
AI summary
The technology described includes methods for pretraining a document encoder model based on multimodal self cross-attention. One method includes receiving image data that encodes a set of pretraining documents. A set of sentences is extracted from the image data. A bounding box for each sentence is generated. For each sentence, a set of predicted features is generated by using an encoder machine-learning model. The encoder model performs cross-attention between a set of masked-textual features for the sentence and a set of masked-visual features for the sentence. The set of masked-textual features is based on a masking function and the sentence. The set of masked-visual features is based on the masking function and the corresponding bounding box. A document-encoder model is pretrained based on the set of predicted features for each sentence and pretraining tasks. The pretraining tasks includes masked sentence modeling, visual contrastive learning, or visual-language alignment.