AI Schema Matching Using Two-Stage Table and Attribute Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Schema matching in large-scale databases faces challenges due to heterogeneous data types, inconsistencies in data representation, and the dynamic nature of data, leading to computational inefficiencies and a lack of scalability in traditional methods.
Innovation Solution
A two-stage schema matching approach using generative artificial intelligence (AI) that first filters metadata for table-level matching and then performs attribute-level matching using sample data, leveraging large language models (LLMs) to enhance accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional schema matching methods are used to handle heterogeneous data types and inconsistencies, then comprehensive data comparison is achieved, but computational efficiency deteriorates and scalability is reduced
Solution Approach 1:
The patent divides the schema matching process into two distinct stages: (1) table-level matching that compares metadata and schema structures to identify candidate matching tables, and (2) attribute-level matching that performs detailed column comparisons only on identified candidates. This segmentation reduces the overall computational complexity by avoiding exhaustive pairwise comparisons of all attributes across all tables.
Solution Approach 2:
The patent performs preliminary filtering at the table level by comparing metadata, table names, and schema structures before proceeding to attribute-level comparison. This preliminary action identifies a smaller subset of candidate matching tables, thereby reducing the number of computationally intensive attribute comparisons needed in the subsequent stage.
2Measurement precision
If exhaustive schema comparison is performed across all tables and attributes, then matching precision is improved, but the complexity of large-scale databases becomes unmanageable
Solution Approach 1:
The patent segments the schema matching task into table-level and attribute-level operations. The table-level stage handles the complexity of identifying candidate matches through metadata comparison, while the attribute-level stage focuses on precise column matching only for identified candidates. This segmentation makes the system manageable for large-scale databases by breaking down the overwhelming task into smaller, more tractable subtasks.
Solution Approach 2:
The patent introduces metadata as an intermediary layer that facilitates table-level matching without requiring direct attribute-by-attribute comparison of all tables. By using metadata (table names, data types, constraints) as a mediator, the system can efficiently identify candidate matching tables before performing detailed attribute comparisons, thereby reducing system complexity.
3Adaptability or versatility
If dynamic data changes are accommodated in real-time, then adaptability of schema matching is improved, but computational overhead increases
Solution Approach 1:
The patent performs preliminary table-level matching using metadata before detailed attribute comparison. When data changes occur, this preliminary filtering step can be quickly re-executed to identify new candidate matches, allowing the system to adapt to dynamic data changes without performing exhaustive attribute comparisons from scratch, thereby controlling computational overhead.
Solution Approach 2:
The two-stage schema matching approach can be periodically re-executed or triggered on-demand when data changes are detected. The computationally lighter table-level stage can be performed frequently to detect changes, while the more intensive attribute-level stage is performed only when necessary, creating a periodic action pattern that balances adaptability with computational efficiency.
Data Source
AI summary
A computer-implemented method for improved schema matching of two databases is disclosed. The method can receive a schema of a source table from a first database and a schema of a plurality of target tables from a second database, identify one or more matching tables among the plurality of target tables based on comparison of the schema of the source table and the schema of the plurality of target tables using a large language model, obtain first sample attribute data from the source table and second sample attribute data from a selected matching table, and identify one or more pairs of matching attributes between the source table and the selected matching table based on comparison of the first sample attribute data and the second sample attribute data using the large language model. Related systems and software for implementing the method are also disclosed.