Multi-Modal Multi-Granular Document Extraction Model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning models for extracting information from PDF documents are inefficient and inaccurate as they are typically trained on a single level or granularity, failing to effectively extract information across different granularities and modalities, leading to suboptimal results and the need for multiple models, which increases costs and resource consumption.
Innovation Solution
A multi-modal multi-granular model that analyzes document regions of various sizes and granularities, incorporating textual and visual features through self-attention and cross-attention mechanisms, generating transformed feature vectors for use in multiple tasks such as document classification and entity recognition, thereby reducing the need for separate models and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single machine learning model is used to extract information from PDF documents, then the device complexity is reduced, but the measurement precision and reliability of information extraction deteriorate because the model cannot effectively capture inter-region information and relationships across different granularities
Solution Approach 1:
The patent merges multiple single-granularity models into a single multi-granularity model that simultaneously processes page-level, region-level, and token-level information. This unified model integrates features from different granularities and modalities (visual, textual, layout) to achieve accurate information extraction without requiring separate models for each granularity level.
Solution Approach 2:
The multi-granularity model is designed to perform multiple functions: it can extract information at different granularities (page, region, token), process multiple modalities (visual, textual, layout), and identify relationships across regions. This universal model replaces multiple specialized models while maintaining or improving extraction accuracy.
2Measurement precision
If multiple machine learning models are used to extract information from different regions of PDF documents, then the measurement precision improves, but the device complexity and training maintenance costs increase
Solution Approach 1:
The patent combines multiple specialized models into a single multi-granularity model that handles different regions and granularities within one unified architecture. This reduces the number of models from multiple to one, simplifying the system while maintaining the precision benefits of multi-region analysis.
Solution Approach 2:
The unified model is designed to be universal, capable of processing various document regions (headers, tables, paragraphs, images) at different granularities (page, region, token levels) and multiple modalities, replacing the need for multiple specialized models.
3Device complexity
If conventional single-granularity models are used for information extraction, then the device complexity is low, but the productivity and efficiency deteriorate due to the need for multiple separate models and increased maintenance requirements
Solution Approach 1:
The patent merges multiple processing functions into a single multi-granularity model that can simultaneously analyze documents at page, region, and token levels, eliminating the need to run multiple separate models and improving processing efficiency.
Solution Approach 2:
The universal multi-granularity model performs multiple extraction tasks in one pass, including entity recognition, relationship extraction, and classification across different granularities, thereby increasing productivity without proportionally increasing complexity.
4Device complexity
If single-modality models are used for document analysis, then the device complexity is reduced, but the measurement precision deteriorates because inter-modal relationships are ignored
Solution Approach 1:
The patent merges multiple modality processing capabilities into a single multi-modal model that simultaneously analyzes visual features, textual features, and layout features, capturing their interrelationships to improve extraction accuracy.
Solution Approach 2:
The multi-modal model is designed to process multiple types of features (visual from CNN, textual from OCR, layout structures) within a unified architecture, enabling it to leverage cross-modal relationships for more accurate information extraction.
Data Source
AI summary
Embodiments are provided for facilitating multimodal extraction across multiple granularities. In one implementation, a set of features of a document for a plurality of granularities of the document is obtained. Via a machine learning model, the set of features of the document are modified to generate a set of modified features using a set of self-attention values to determine relationships within a first type of feature and a set of cross-attention values to determine relationships between the first type of feature and a second type of feature. Thereafter, the set of modified features are provided to a second machine learning model to perform a classification task.