Table Detection Using Semantic Grouping and White Space Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated table recognition systems are inefficient and inaccurate in detecting and extracting unstructured tables from electronic documents, especially those with OCR errors and unique formats, due to their reliance on manual curation, limited training data, and poor performance with abnormal medical results.
Innovation Solution
A system and method that uses semantic grouping of table header terms to identify headers and apply data zone grouping techniques, such as white space correlation, to align columns and cleanse data, allowing for the detection and extraction of tables in various formats, including those with errors, and flagging abnormal results with a confidence factor.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated table recognition systems use image pattern matching software to look for established patterns of data elements, then the processing speed is improved, but the accuracy deteriorates when detecting unstructured tables with different formats
Solution Approach 1:
The system changes the parameter of table format representation from fixed image patterns to flexible semantic structures. By representing tables as sequences of tokens with semantic relationships rather than rigid visual patterns, the system can adapt to various table formats while maintaining processing efficiency through programmatic manipulation of these semantic representations.
Solution Approach 2:
The patent implements dynamic table structure representation where the system can adapt its parsing and recognition approach based on the specific format characteristics of each table encountered. The semantic structure allows the system to dynamically adjust to different table layouts, column arrangements, and formatting styles rather than being constrained by pre-established static patterns.
2Productivity
If automated table recognition systems use training data from example tables, then the recognition speed for known formats is improved, but the adaptability deteriorates for new or unique table formats
Solution Approach 1:
The system employs a universal semantic structure that can represent multiple table formats through a common framework. By defining tables in terms of fundamental semantic elements (headers, data cells, relationships) rather than format-specific patterns, the system achieves multi-functionality that allows it to handle both known and novel table formats with the same core recognition logic.
Solution Approach 2:
The patent transforms the approach from learning format-specific parameters through training to using parameter-free semantic parsing. The system changes how tables are represented by focusing on semantic relationships and structural patterns that are invariant across formats, eliminating the need for retraining when encountering new table styles while maintaining rapid recognition through efficient semantic matching.
3Ease of operation
If OCR software is used to transform PDF image files into text, then the accessibility of table data is improved, but the accuracy deteriorates due to word misspellings, garbage character insertion and text deletion
Solution Approach 1:
The system implements feedback mechanisms where the semantic structure of the table guides the OCR process. By using the expected semantic patterns and structural constraints of tables as feedback, the system can identify and correct OCR errors such as misspellings and garbage characters. The semantic framework provides context that helps distinguish valid table content from OCR artifacts.
Solution Approach 2:
The patent converts the harmful effect of OCR errors into a beneficial filtering process. By establishing semantic rules and structural expectations for valid table content, the system can identify OCR-generated garbage characters and errors as anomalies that deviate from expected semantic patterns. This allows the system to automatically filter out or correct errors, turning the presence of potential errors into an opportunity for validation and cleanup.
Data Source
AI summary
A method, system, and apparatus are provided for processing tables embedded within documents wherein a first table header is detected by using semantic groupings of table header terms to identify a minimum number of table header terms in a scanned line of an text document; a potential data zone is extracted by applying white space correlation analysis to a portion of the text document that is adjacent to the first table header; one or more data zone columns from the potential data zone are grouped and aligned with a corresponding header column in the first table header to form a candidate table; data cleansing is performed on the candidate table; and then one or more columns of the candidate table are evaluated using natural language processing to apply a specified table analysis.