Automatic Complex Table Transformation to Structured Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for extracting data from complex tables in electronic documents are manual, error-prone, and inefficient, as they struggle to distinguish between data values, attribute names, and metadata, and are not designed for querying, leading to unsuccessful data integration.
Innovation Solution
A computer-implemented method that uses semantic technologies like RDF/SPARQL and database dependencies to automatically convert compact tables into schema-less searchable data, enabling query answering with minimal manual effort by transforming tables into relational-like layouts and utilizing relational dependencies for complex query execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual methods are used to extract data from complex tables, then data integration can be performed, but the process is highly manual, error-prone, and unsuccessful for complex table formats
Solution Approach 1:
The system automatically transforms complex table layouts into relational database schemas without requiring manual intervention. The transformation process self-adapts to different table formats by analyzing table structure patterns, identifying headers, and mapping relationships automatically, eliminating the need for manual data integration efforts.
Solution Approach 2:
The system changes the structural parameters of table data by transforming compact table layouts into normalized relational schemas. This involves converting table cells into rows and columns, identifying primary keys and foreign keys, and reorganizing data relationships to match database schema requirements, thereby enabling automated processing.
2Area of stationary object
If table layouts are designed to save space in documents, then document compactness is improved, but the layouts are not designed for searching queries and data extraction becomes difficult
Solution Approach 1:
The system segments compact table layouts into discrete relational database tables by identifying table boundaries, headers, and data regions. This segmentation process separates the compact visual representation into structured relational components that can be independently queried and manipulated in the database system.
Solution Approach 2:
The system introduces an intermediate transformation layer that converts compact table representations into relational database schemas. This intermediary process acts as a mediator between the space-efficient document format and the query-friendly database format, automatically mapping table structures without requiring manual intervention.
3Productivity
If automated transformation is implemented, then productivity is improved, but device complexity increases due to semantic technologies and dependency tracking
Solution Approach 1:
The system employs a universal transformation framework that handles multiple table formats and layouts through a single automated process. The semantic transformation engine and dependency tracking mechanisms serve multiple functions including schema generation, relationship identification, and query optimization, reducing the need for separate specialized systems for different table types.
4Measurement precision
If dependency information is maintained for all dependent objects, then query accuracy is improved, but storage requirements and processing overhead increase
Solution Approach 1:
The system maintains dependency information selectively rather than comprehensively for all possible objects. The dependency tracking mechanism focuses on maintaining relationships that are relevant to query operations, storing only the necessary foreign key relationships and referential constraints needed for accurate query results, thereby reducing storage overhead while preserving query accuracy.
Data Source
AI summary
An information processing system, a computer readable storage medium, and a computer-implemented method, collect tables from a corpus of documents, convert the collected tables to flattened table format and organized to be searchable by schema-less queries. A method collects tables, extracts feature values from collected table data and collected table meta-data for each collected table. A table classifier classifies each collected table as being a type of table. Based on the classifying, the collected table is converted to a flattened table including table values that are the table data and the table meta-data of the collected table. Dependencies of the data values are mapped. The flattened table and mapped dependencies are stored in a triple store searchable by schema-less queries. The table classifier learns and improves its accuracy and reliability. Dependency information is maintained among a plurality of database tables. The dependency information can be updated at variable update frequency.