Dataset Join Candidate Identification via Pre-computed Independence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud computing environments, determining the joinability of datasets is time-consuming and resource-intensive, particularly when dealing with large numbers of datasets and fields, as current methods require actual joining and comparison of entries to assess dependence, leading to significant delays and discouragement for users.
Innovation Solution
A system that estimates the independence of datasets by pre-calculating occurrence and probability values for entry combinations, using a G-test and chi-squared distribution, and storing these values for quick identification of join candidates, allowing users to suggest viable join candidates without the need for actual joining, thereby reducing user wait time and computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If actual joining and comparison of dataset entries is performed to assess dependence, then measurement precision of joinability is improved, but time consumption and computational resources increase significantly
Solution Approach 1:
The system performs preliminary actions by pre-calculating occurrence counts and probability values for all possible entry combinations across datasets before any join candidate assessment is needed. These pre-computed statistics are stored and reused to quickly evaluate joinability without performing actual dataset joins at query time, thus resolving the contradiction between accurate measurement and time consumption.
Solution Approach 2:
Instead of performing actual dataset joins to assess dependence, the system creates simplified statistical copies (occurrence counts and probability values) that represent the essential joinability characteristics. These statistical copies enable rapid evaluation while maintaining sufficient accuracy for identifying viable join candidates.
2Reliability
If actual joining of datasets is performed to identify join candidates, then reliability of join candidate identification is improved, but computational resources and processing time increase
Solution Approach 1:
The system extracts only the essential statistical features (occurrence counts and probability values) needed for joinability assessment from the actual datasets, separating these critical metrics from the full dataset content. This extraction enables reliable join candidate identification using minimal computational resources, as only the pre-computed statistics need to be compared rather than performing complete dataset joins.
3Productivity
If pre-calculated occurrence and probability values are used to identify join candidates, then productivity is improved, but device complexity increases
Solution Approach 1:
The system performs preliminary calculations of occurrence counts and probability values for all entry combinations and stores them in accessible structures. This pre-computation phase, while adding some architectural complexity, enables extremely fast query-time evaluation where join candidates are identified by simple comparison of pre-stored statistical values rather than complex real-time computations.
Solution Approach 2:
The system introduces statistical intermediaries (occurrence counts and probability values) that mediate between the raw datasets and the join candidate identification process. These intermediary statistical structures simplify the overall system architecture by decoupling the complex dataset storage from the evaluation logic, enabling high productivity through efficient statistical comparison.
Data Source
AI summary
Examples are described herein that relate to determining a level of relatedness between datasets. An approximation can be made of whether an entry in a first dataset appears in a same row as an entry in a second dataset. The approximation can be made by grouping entries in the second dataset together and determining an occurrence that an entry occurs in a same row as any of the entries in a grouping of entries. A test of independence between datasets can be made based at least on the occurrence values. Datasets can be ranked according to level of independence and presented to a user as candidates to join with a dataset. Occurrence values or rankings can be precomputed and available for use so that join candidates can be presented with a little amount of perceived delay to a user. A user interface can present join candidates for a dataset and allow the user the select datasets for joining. Joining of first and second datasets can supplement entries in both of the datasets and create a third dataset.