Dataset Joining via ML Column and Row Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently joining datasets from different platforms due to variations in data organization, formatting, and annotation, which can lead to errors and failures in data conversion and merging processes.
Innovation Solution
A computer-implemented method using trained machine-learning models for column and row discovery, generating vector embeddings and approximate nearest neighbor indices to create column and row maps that index corresponding columns and rows across datasets, facilitating their combination into a unified dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional import functions are used to convert data between platforms, then data can be transported between different platforms, but errors and failures occur due to platform updates and proprietary internal details
Solution Approach 1:
The patent introduces an intermediary data representation layer that translates proprietary platform-specific data formats into a universal intermediate format. This mediator layer handles the conversion between different database platforms (Oracle, SQL Server, MySQL, etc.) without directly relying on proprietary internal details, thereby maintaining reliability across platform updates while preserving adaptability to multiple platforms
Solution Approach 2:
The patent segments the data conversion process into distinct layers: data extraction from source platform, transformation to intermediate representation, and loading to target platform. This segmentation isolates platform-specific operations from the core conversion logic, allowing individual platform adapters to be updated without affecting the overall system reliability
2Adaptability or versatility
If datasets from different platforms are joined using traditional methods, then data integration can be achieved, but the process becomes complex and error-prone due to different data organization and formatting
Solution Approach 1:
The patent applies homogeneity by standardizing diverse data formats from different platforms into a unified intermediate representation. All datasets are transformed to share common structure, naming conventions, and formatting rules, which simplifies the joining process by eliminating the need for complex platform-specific matching logic and reduces errors from format mismatches
3Quantity of substance
If datasets with different annotations and organization are joined, then comprehensive data consolidation can be achieved, but matching corresponding elements becomes challenging
Solution Approach 1:
The patent uses an intermediary semantic layer that maps diverse annotations and organizational structures to a common vocabulary. This mediator performs semantic matching by translating different data element names and structures into equivalent representations, enabling accurate matching of corresponding elements across datasets with different annotations while maintaining complete data consolidation
Data Source
AI summary
A computer-implemented method comprising: receiving, with a computer, first and second datasets; performing, with the computer, column discovery on the first and second datasets using a first trained machine-learning model to produce a column map that indexes one or more columns in the first dataset to one or more columns in the second dataset; performing, with the computer, row discovery on the first and second datasets using a second trained machine-learning model, a trained approximate nearest neighbor index, and the column discovery to produce a row map that indexes one or more rows in the first dataset to one or more rows in the second dataset; combining, with the computer, the first and second datasets using the column map and the row map to form a combined dataset; and performing one or more actions with the combined dataset.