Database Schema Matching with Retrieval-Enhanced LLMs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Schema matching in data integration is challenging due to textual and semantic heterogeneity, requiring manual effort and prone to errors, and existing machine-learning solutions lack accuracy and require access to source schema data.
Innovation Solution
A system and method using retrieval-enhanced large language models (LLMs) to represent database schemas as passage-based structured documents, leveraging retrieval mechanisms to identify semantically similar attributes across schemas, reducing the need for manual intervention and improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If machine-learning-based solutions are used for schema matching, then automation is improved, but accuracy deteriorates and manual mapping is still required
Solution Approach 1:
The system uses retrieval-enhanced large language models to automatically perform schema matching without requiring manual mapping or training data. The model retrieves relevant schema information from the source database and autonomously generates accurate matchings, eliminating the need for human intervention while maintaining high accuracy.
Solution Approach 2:
The patent replaces traditional machine-learning models that require manual training with retrieval-enhanced large language models that use contextual retrieval and generation. This substitution eliminates the need for manual mapping while improving accuracy through contextual understanding of schema elements.
2Extent of automation
If existing machine-learning solutions are used, then automation is improved, but access to source schema data is required which may be unavailable due to privacy concerns
Solution Approach 1:
The system introduces retrieval-enhanced large language models as an intermediary that processes schema information without requiring direct access to the source schema data. The model retrieves relevant information through contextual queries and generates matchings based on this retrieved information, maintaining automation while respecting data privacy constraints.
3Measurement precision
If manual schema matching is performed, then accuracy is improved, but time consumption increases and scalability deteriorates
Solution Approach 1:
The retrieval-enhanced large language model performs schema matching autonomously by retrieving relevant schema information and generating matchings without human intervention. This self-service approach eliminates time-consuming manual processes while maintaining high accuracy through contextual understanding and intelligent retrieval mechanisms.
Data Source
AI summary
A method, computer program product, and computing system for matching database schemas to align attributes in corresponding tables. A first attribute of a source table structured according to a source schema is represented in a document format and a plurality of target tables structured according to a target schema are represented in the document format. The document format of each of the plurality of target tables are searched to retrieve a set of target tables that include second attributes corresponding to the first attribute based on semantic similarity between the first attribute and the plurality of target tables, and a set of second attributes is selected from the retrieved set of target tables that are similar to the first attribute using a large language model (LLM). A ranked list of second attributes that match the first attribute is then generated.