Auto-join System for Tables with Different Representations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to efficiently join tables with different representations in ad-hoc data analysis scenarios, as they require extensive data cleansing and preparation, making them slow and expensive for one-off data analysis tasks.
Innovation Solution
The auto-join system determines joinable row pairs, learns a transformation model, transforms values in one table, and uses a fuzzy join process to handle inconsistencies, enabling efficient equi-joining of tables with different representations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional join methods are used to connect tables with different representations, then join accuracy can be maintained, but processing time increases and productivity decreases
Solution Approach 1:
The system performs preliminary actions by automatically detecting joinable row pairs and learning transformation models from sample data before executing the full join operation. This pre-processing step enables the system to establish transformation rules upfront, allowing rapid application of these rules to the entire dataset without sacrificing join accuracy.
Solution Approach 2:
The system employs self-service mechanisms through automated transformation learning that discovers and applies transformation rules autonomously without manual intervention. The fuzzy match detection and transformation model learning processes operate automatically, enabling the system to serve itself in identifying and executing appropriate join strategies for tables with different representations.
2Reliability
If extensive data cleansing and preparation are performed to join tables with different representations, then join reliability improves, but time consumption increases
Solution Approach 1:
The system replaces manual data cleansing and preparation mechanics with automated transformation learning and fuzzy match detection. Instead of requiring users to manually clean and prepare data for joining, the system automatically learns transformation models from the data itself, substituting mechanical data preparation processes with intelligent automated discovery and application of transformation rules.
Solution Approach 2:
The system applies parameter changes by dynamically adjusting transformation parameters based on learned models from sample row pairs. Rather than using fixed data preparation procedures, the system adapts transformation parameters automatically to match the specific characteristics of the tables being joined, maintaining reliability while reducing preparation time.
3Manufacturing precision
If manual transformation and joining processes are used, then transformation precision can be controlled, but automation level remains low and operational complexity increases
Solution Approach 1:
The system achieves self-service through automated transformation learning that discovers transformation rules autonomously from sample data. The system automatically detects joinable row pairs, learns transformation models, and applies these transformations without manual intervention, thereby increasing automation level while maintaining transformation precision through systematic learning processes.
Solution Approach 2:
The system implements feedback mechanisms by using detected joinable row pairs to learn and refine transformation models. The automated process continuously improves transformation precision through feedback from successful match detections, allowing high automation levels to coexist with controlled transformation precision through iterative learning and adaptation.
Data Source
AI summary
Methods and systems for joining two tables are provided. At least two tables to be joined are received. A joinable row pair between the at least two tables is determined. The determined joinable row pair includes a first row from a first table having a common string value with a second row from a second table of the at least two tables. A transformation model is generated from the determined joinable row pair. A column of the first table is transformed based on the generated transformation model. The transformed first table is joined with the second table.