Document Image Semantic Graphs for Visually Rich Layout Understanding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document processing technologies fail to comprehensively consider the layout relationships and multi-granularity, multi-modality information of visually-rich documents, leading to inadequate understanding of their content.
Innovation Solution
A method and apparatus that acquire text and visual nodes of multiple granularities, construct a multi-granularity and multi-modality document graph, and determine semantic feature information using a semantic feature extraction model to represent the document's rich information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional document processing methods are used, then processing speed is maintained, but understanding accuracy of visually-rich documents is insufficient
Solution Approach 1:
The patent segments the document processing task into multiple granularities (page level, block level, element level) and processes text and visual elements separately through different encoders, then integrates them through cross-granularity attention mechanisms. This segmentation allows the system to capture detailed information at each level while maintaining overall document understanding accuracy.
Solution Approach 2:
The patent introduces a multi-granularity dimension by representing documents at different levels of abstraction (page, block, element) simultaneously. This dimensional expansion allows the model to consider both fine-grained text details and coarse-grained layout structures, significantly improving understanding accuracy without linearly increasing complexity.
2Measurement precision
If multi-granularity and multi-modality information is comprehensively considered, then understanding accuracy is improved, but processing complexity increases
Solution Approach 1:
The patent divides the complex processing task into separate text encoding and visual encoding pathways, each handling specific modalities at different granularities. This segmentation allows independent optimization of each pathway while reducing the overall computational complexity compared to a monolithic approach.
Solution Approach 2:
The patent introduces cross-granularity attention mechanisms as intermediary components that bridge text and visual representations at different levels. These attention mechanisms efficiently integrate multi-granularity and multi-modality information without requiring direct complex interactions between all elements, thus managing processing complexity.
3Loss of information
If layout relationships and multi-granularity information are considered, then semantic feature richness is improved, but computational resources required increase
Solution Approach 1:
The patent segments semantic feature extraction into hierarchical levels (page-level semantics, block-level semantics, element-level semantics), allowing the system to extract and process semantic information at appropriate granularities. This prevents unnecessary computation at all levels simultaneously, reducing overall computational resource consumption while maintaining semantic feature richness.
Solution Approach 2:
The patent adds a granularity dimension to semantic representation, enabling the model to capture semantic features at multiple scales. This dimensional approach allows efficient computation by processing coarse-grained semantics first and refining to fine-grained details only where necessary, optimizing computational resource usage.
Data Source
AI summary
In a method for processing a document image, a document image to be processed is acquired. Text nodes of multiple granularities, visual nodes of multiple granularities, respective node information of the text nodes, and respective node information of the visual nodes in the document image are obtained. A multi-granularity and multi-modality document graph is construct based on the text nodes of multiple granularities, the visual nodes of multiple granularities, the respective node information of the text nodes and the respective node information of the visual nodes. Multi-granularity semantic feature information of the document image is determined based on the multi-granularity and multi-modality document graph, the respective node information of the text nodes and the respective node information of the visual nodes.