Table Data Extraction Using Cell Classification and Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Regulatory content documents with tables often vary in size and layout, making it difficult for computers to extract relevant data, as the format and layout are diverse, and existing methods struggle to identify relevant requirements effectively.
Innovation Solution
A computer-implemented method and system that processes image data of tables to generate tokens, language embeddings, and position embeddings, associating tokens with cells and using a cell classifier to determine relationships between cells based on these embeddings, facilitating the extraction of data in a structured machine-readable format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional methods are used to extract data from tables, then the process is simple, but the accuracy and reliability of data extraction deteriorates due to varying formats and layouts
Solution Approach 1:
The patent segments the table processing into distinct components: image data processing, token generation, embedding generation (language and position), cell classification, and relationship identification. This segmentation allows each component to specialize in handling specific aspects of table data extraction, improving overall reliability while managing complexity through modular architecture.
Solution Approach 2:
The patent introduces intermediary representations including tokens, language embeddings, and position embeddings that mediate between the raw image data and the final data extraction. These intermediaries transform the complex visual data into structured representations that can be systematically processed and classified, enhancing extraction accuracy without directly exposing the complexity of the processing system.
2Loss of information
If comprehensive table processing is performed to maintain format and layout, then data extraction completeness improves, but processing time increases
Solution Approach 1:
The patent performs preliminary actions by pre-processing the image data to generate tokens and their corresponding language and position embeddings before the actual cell classification and relationship identification. This preliminary processing organizes the data in advance, allowing the subsequent classification and relationship determination to proceed more efficiently, thus reducing overall processing time while maintaining completeness.
Solution Approach 2:
The patent transforms the two-dimensional image data into multiple dimensional representations through tokenization, language embeddings, and position embeddings. This dimensional transformation allows the system to process and analyze table data more efficiently by representing it in a structured, multi-dimensional space that facilitates faster classification and relationship identification while preserving all original information.
3Measurement precision
If detailed cell classification is performed to identify relationships, then the precision of requirement identification improves, but the ease of operation deteriorates
Solution Approach 1:
The patent implements self-service through automated cell classification and relationship identification processes that operate without requiring manual intervention. The system automatically analyzes cell classifications, position embeddings, and generated relationships to identify relevant requirements, eliminating the need for manual operation while maintaining high precision in requirement identification.
Data Source
AI summary
A computer-implemented method for extracting data from a table comprising a plurality of cells is provided. The method comprises: receiving image data representing a physical layout of the table and words or characters in the table; generating a plurality of tokens, each token representing a semantic unit of the words or characters; and generating at least one language embedding and at least one position embedding associated with each token. The method further comprises: associating at least one token with a cell; receiving the tokens at an input of a cell classifier trained to generate a cell classification for each cell based on the language embeddings and the position embeddings of the at least one token associated with the cell; and determining a relationship exists between at least two cells based at least in part on the cell classifications and the position embeddings of the tokens.