Database Table Merging via Classification Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Merging two or more database tables is a time-consuming and burdensome process, often requiring significant manual input and human intervention, especially when dealing with redundant records, and existing techniques like ETL processes are inefficient.
Innovation Solution
A computing system that dynamically merges database tables based on user-specified parameters, such as desired fields and confidence levels, using a classifier to determine record matches and preserve original tables for generating various merged versions, while maintaining links to relational tables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional ETL processes are used to merge database tables, then merging can be completed, but the process is time-consuming and requires significant manual input
Solution Approach 1:
The system performs self-service by automatically comparing records from different database tables using machine learning classifiers to identify matches and merges records without requiring manual human intervention. The classifier autonomously determines which records represent the same entity based on learned patterns from training data, eliminating the need for manual review and significantly reducing merging time while maintaining high accuracy.
2Measurement precision
If manual merging processes are used, then record matching can be accurate, but the process becomes burdensome and requires significant human intervention
Solution Approach 1:
The patent replaces manual mechanical operations with an automated machine learning system. The classifier uses computational algorithms to compare records across database tables, automatically identifying matches based on multiple attributes and confidence scores. This substitution eliminates the need for manual record-by-record review while maintaining or improving matching accuracy through consistent, repeatable automated comparisons.
Solution Approach 2:
The machine learning classifier acts as an intermediary between the raw database records and the final merged output. It processes records through trained models that evaluate multiple attributes simultaneously, generating confidence scores that mediate the decision-making process. This intermediary layer automates the complex judgment required for accurate record matching while reducing operational burden on users.
3Quantity of substance
If database tables are merged to eliminate redundancy, then data efficiency improves, but the merging process becomes complex and time-consuming
Solution Approach 1:
The system changes parameters by using confidence score thresholds to automatically control the merging process. The classifier generates confidence scores for each potential record match, and predefined threshold parameters automatically determine which matches proceed to merging. This parameter-based approach simplifies the complex merging process by replacing manual judgment with automated threshold comparisons, reducing process complexity while effectively eliminating data redundancy.
Data Source
AI summary
The present disclosure relates to merging database tables. Systems and methods may involve performing a comparison between the first set of records and the second set of records and identifying a plurality of record pairs based on the comparison. Each record pair may comprise a record in the first set of records and a record in the second set of records. In addition, A feature signature may be generated for each record pair by comparing field values in each record pair. The feature signature may be classified to identify at least one related record pair. A merged database table may be generated such that it comprises the at least one related record pair and comprises a set of unique records among selected from the first set of records and the second set of records.