Automated Data Relationship Discovery for Heterogeneous Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big data systems face challenges in analyzing data due to noise, duplication, and heterogeneity, requiring substantial manual processes for data cleaning and curation, which become impractical as data volumes increase, and struggle to merge datasets from disparate sources effectively.
Innovation Solution
A computer-based system implements algorithms for data relationship discovery, enabling the merging and enrichment of datasets from different sources by generating visual representations of potential joins and using scoring functions to recommend optimal column pairs for joining, thereby automating the data preparation and analysis process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual data cleaning and curation processes are used, then data quality can be improved, but the process cannot scale as data volumes increase
Solution Approach 1:
The system performs automated data cleaning, validation, and transformation without requiring manual intervention. The automated ETL pipeline independently processes data from multiple sources, applies predefined cleaning rules, and prepares data for analysis, eliminating the need for manual scaling while maintaining quality standards.
Solution Approach 2:
Manual mechanical data cleaning processes are replaced with automated computational algorithms. The system uses programmatic validation rules, automated error detection, and machine learning models to clean and curate data at scale, substituting human manual operations with automated mechanical systems.
2Quantity of substance
If datasets from disparate sources are merged, then data enrichment is achieved, but heterogeneity and size make merging challenging
Solution Approach 1:
The merging process is divided into modular stages: data ingestion, validation, transformation, and loading. Each stage handles specific aspects of heterogeneity separately, making the overall complex process manageable. The system segments data from different sources into standardized formats before merging, reducing complexity.
Solution Approach 2:
The system transforms data from disparate sources by changing their parameters to standardized formats. Data types, formats, and schemas are normalized through parameter transformation, allowing heterogeneous data to be merged into a unified structure without increasing complexity.
3Speed
If data is analyzed without proper formatting, then processing speed increases, but analysis results become useless due to noise
Solution Approach 1:
Data formatting and cleaning are performed as preliminary actions before analysis. The automated ETL pipeline pre-processes data by validating formats, removing errors, and standardizing structures in advance, so that when analysis occurs, the data is already prepared and ready, maintaining both speed and reliability.
Data Source
AI summary
The present disclosure related to techniques for analyzing data from multiple different data sources to determine a relationship between the data (also referred to herein a “data relationship discovery”). The relationships between any two compared datasets may be used to determine one or more recommendations for merging (e.g., joining), or “blending,” the data sets together. Relationship discovery may include determining a relationship between a subset of data, such as a relationship between a pair of columns, or column pair, each column in a different dataset of the datasets that are compared. Given two datasets to process for relationship discovery, relationship discovery may identify and recommends a ranked subset of column pairs between two compared datasets. The ranked column pairs identified as a relationship may be useful for blending the datasets with respect to those column pairs.