Semantic Data Joining via Statistical Co-occurrence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional join processing in databases is inefficient and inconvenient, especially when data sets have different representations, as it requires manual intervention to find intermediate mapping tables, and existing systems struggle to perform joins automatically in such scenarios.
Innovation Solution
The system predicts semantic relationships between data sets using statistical co-occurrence scores based on historical data, enabling automated semantic joins without the need for intermediate mapping tables by identifying pairs of values and calculating aggregate correlation scores to infer join relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional join processing is used with string equality comparisons, then join accuracy is maintained for curated data, but manual intervention is required to find intermediate mapping tables and the process becomes time-consuming
Solution Approach 1:
The system performs self-service by automatically computing statistical co-occurrence scores between values in different data sets and inferring semantic relationships without requiring manual intervention to find mapping tables. The processor autonomously identifies pairs of values, calculates their co-occurrence statistics, and determines semantic relationships, eliminating the time-consuming manual process while maintaining join accuracy through statistical analysis.
2Adaptability or versatility
If manual join processing is performed for data sets with different representations, then semantic relationships can be established, but the process becomes complex and infeasible under some circumstances
Solution Approach 1:
The patent replaces the mechanical manual process of finding and applying mapping tables with an automated statistical computing system. The processor automatically computes statistical co-occurrence scores for pairs of values from different data sets and infers semantic relationships, substituting the complex manual mechanical process with an automated computational approach that handles diverse representations systematically and efficiently.
3Adaptability or versatility
If intermediate mapping tables are used for semantic joins, then joins between different representations are enabled, but the requirement for multiple mapping tables increases system complexity
Solution Approach 1:
The patent extracts the essential semantic relationship information directly from the data sets by computing statistical co-occurrence scores between pairs of values, eliminating the need for separate intermediate mapping tables. Instead of requiring multiple mapping tables to bridge different representations, the system directly analyzes the statistical relationships within the data itself, extracting the necessary join information without auxiliary structures.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Examples of the disclosure enable performing semantic joins using a big table corpus. Pairs of values from at least two data sets are identified. The pairs of values include one value from a first one of the data sets and one value from a second one of the data sets. Statistical co-occurrence scores for the identified pairs of values are determined based on historical co-occurrence data. The determined statistical co-occurrence scores are used for predicting a semantic relationship between the at least two data sets. The predicted semantic relationship is used for joining the at least two data sets.