Automated Dataset Joining via Column Profiling and Visualization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big data systems face challenges in analyzing data due to noisy, erroneous, and duplicate data, requiring substantial manual processes for cleaning and curation, which become impractical as data volume increases, and struggle with merging datasets from disparate sources with different formats and structures.
Innovation Solution
A computer-based system implements data relationship discovery techniques to identify and visualize potential joins between datasets, using algorithms and graphical interfaces to recommend types of joins such as left, right, or outer joins, enabling the merging of seemingly unrelated data sources for enrichment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual processes are used to clean and curate data, then data quality can be improved, but the process cannot scale as data volume increases
Solution Approach 1:
The system performs automated data cleaning and curation without manual intervention. The computer system automatically ingests data from multiple sources, profiles columns to identify relationships, and generates join recommendations, allowing the data processing system to serve itself rather than requiring manual operations for each dataset.
Solution Approach 2:
Manual mechanical processes for data cleaning are replaced with automated computational algorithms. The system uses column profiling algorithms, relationship discovery algorithms, and automated join generation to substitute human manual operations with machine-based automated processes that can scale to large data volumes.
2Adaptability or versatility
If data from disparate sources with different formats is merged, then data enrichment is achieved, but the complexity of finding similarities and joining columns increases
Solution Approach 1:
The system introduces an intermediary profiling layer between raw data and join operations. Column profilers extract metadata and characteristics from each dataset's columns, creating an intermediary representation that facilitates relationship discovery. This intermediary profile data serves as a mediator that simplifies the complex task of matching columns across heterogeneous data sources.
Solution Approach 2:
The system changes the parameters of data representation by transforming raw data into profiled metadata. Instead of directly comparing raw data values, the system profiles columns to extract parameters such as data types, null percentages, and relationship indicators, making the joining process more manageable by operating on transformed parameter representations rather than raw data.
3Ease of operation
If large datasets are visualized to help users understand merging options, then user comprehension is improved, but the visualization itself becomes difficult to generate and interpret
Solution Approach 1:
The system extracts only the essential information needed for visualization from large datasets. Instead of visualizing entire datasets, the column profilers extract key metadata and relationship indicators from columns, and the system generates visualizations based on this extracted profile information, making visualization feasible by taking out only the necessary elements.
Solution Approach 2:
The system segments the visualization task by breaking down large datasets into manageable column-level profiles. Instead of attempting to visualize entire large datasets at once, the system divides the data into individual column profiles, analyzes relationships at the column level, and generates visualizations based on these segmented units, making the overall visualization process more manageable.
Data Source
AI summary
Techniques are disclosed for a system that provides an intuitive way for merging or joining data from different datasets. The system may provide graphical interfaces to enable a user to combine or join datasets identified as having a relationship. In at least one embodiment, the system can determine options for joining datasets, such as by a left join, right join, or outer join. A graphical interface may display a visual representation (e.g., a “Glenn Diagram”) illustrate options for joining datasets based on identifying a relationship between the data sets. The representation may further illustrate one or more types of joins and information about the data, such as rows where data may be joined based on the type of join function for the relationship by columns. The visual representation may indicate where the datasets can be joined, such that they are not overlapping.