Table Recognition Using Image and OCR Features for Cell Relationships
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image-based table recognition methods fail to utilize language information from tables, leading to suboptimal extraction of semantic and structural information, and existing tools focus on individual cells without considering relationships between cells.
Innovation Solution
A table recognition model combining image-based and OCR-based components, utilizing a transformer-based model with a vision encoder, split decoder, OCR decoder, and relationship decoder to extract visual, semantic, and relational information from tables, enhanced by pre-training tasks like mask OCR token learning, split denoising, and table rotation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional image-based table recognition methods are used, then the visual structure of tables can be extracted, but language information and semantic meaning from the tables are not utilized, leading to suboptimal results
Solution Approach 1:
The patent combines image-based table recognition with OCR-based semantic extraction into a unified transformer model. The model integrates visual features from the table image with textual features from OCR-recognized words, allowing simultaneous extraction of both structural and semantic information. This merging resolves the contradiction by ensuring language information is not lost while maintaining visual structure extraction capabilities.
Solution Approach 2:
The patent creates a composite feature representation by concatenating image feature vectors with OCR word feature vectors. This composite approach allows the model to leverage both visual and linguistic information streams, improving semantic information extraction while preserving the benefits of image-based structure recognition.
2Ease of operation
If existing tools focus on a single cell of the table, then individual cell detection can be performed, but relationships between cells are ignored, providing sub-optimal results when extracting information from the table
Solution Approach 1:
The transformer model is designed to perform multiple functions simultaneously: it detects individual cells, identifies row and column structures, and extracts relationships between cells. The model processes the entire table as a unified structure rather than isolated cells, enabling it to capture contextual relationships while maintaining ease of cell-level detection through its attention mechanism.
Solution Approach 2:
The patent transitions from cell-level isolated processing to table-level holistic processing by introducing a new dimensional perspective. The model considers the entire table structure and cell relationships in addition to individual cell properties, adding a relational dimension to the detection process that captures inter-cell dependencies.
3Productivity
If a model extracts visual features from table images without using language information, then the processing speed is maintained, but the accuracy of semantic and structural information extraction deteriorates
Solution Approach 1:
The model performs preliminary OCR recognition to extract word-level textual information from the table image before processing. This preliminary action captures language information in advance, allowing the main transformer model to efficiently process both visual and textual features in parallel, maintaining processing speed while improving semantic extraction accuracy through the integration of pre-extracted linguistic data.
Data Source
AI summary
In various examples, a table recognition model receives an image of a table and generates, using a first encoder of the table recognition machine learning model, an image feature vector including features extracted from the image of the table; generates, using a first decoder of the table recognition machine learning model and the image feature vector, a set of coordinates within the image representing rows and columns associated with the table, and generates, using a second decoder of the table recognition machine learning model and the image feature vector, a set of bounding boxes and semantic features associated with cells the table, then determines, using a third decoder of the table recognition machine learning model, a table structure associated with the table using the image feature vector, the set of coordinates, the set of bounding boxes, and the semantic features.