Dataset Joining via Intermediate Mapping Entities
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database technologies face challenges in joining datasets that use different keys to identify data entries, making it difficult to correlate and combine data from multiple databases, especially when direct correlation is not possible due to varying identifier types.
Innovation Solution
A method is introduced to join datasets using an intermediate mapping entity that maps different key types, allowing for the correlation of data entries even when they are identified by distinct keys, such as names and social security numbers, by employing filtering expressions and data structures like Bloom filters and counting filters to facilitate data aggregation across multiple databases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If datasets are merged together to enable querying with multiple expressions, then the ability to answer complex queries is improved, but the complexity of the merging process and data transfer requirements increase
Solution Approach 1:
The patent segments the data merging process into two independent phases: first, local filtering of datasets using boolean expressions to identify relevant records; second, joining of pre-filtered datasets using common keys. This segmentation eliminates the need for complex full-dataset merging while maintaining query capability, as each dataset is independently processed before combination.
Solution Approach 2:
The patent applies preliminary filtering actions to datasets before the actual joining operation. By evaluating boolean expressions and filtering records in advance, the system reduces the volume of data that needs to be merged and transferred, simplifying the subsequent joining process while preserving the ability to answer complex multi-dataset queries.
2Adaptability or versatility
If datasets are merged together to enable querying with multiple expressions, then the ability to answer complex queries is improved, but the amount of data transfer required increases
Solution Approach 1:
The patent segments the data processing workflow into local filtering (performed on individual datasets) and remote joining (performed on filtered results). This segmentation ensures that only the necessary subset of data from each dataset is transferred and combined, dramatically reducing data transfer volume compared to merging entire datasets before querying.
Solution Approach 2:
The patent performs preliminary filtering of datasets using boolean expressions before the joining operation. This preliminary action identifies and retains only the relevant records that satisfy query conditions, eliminating unnecessary data transfer of irrelevant records while maintaining the ability to answer complex queries across multiple datasets.
3Adaptability or versatility
If direct correlation between datasets is attempted using different key types, then the ability to join disparate databases is improved, but the complexity of key mapping increases
Solution Approach 1:
The patent introduces an intermediary key mapping layer that translates between different key types from disparate datasets. This intermediary component receives keys from various datasets, performs type conversion and mapping according to predefined relationships, and outputs compatible keys for joining operations, thereby enabling database compatibility without direct complex key-to-key mapping.
Solution Approach 2:
The patent implements a universal key interface that can accept multiple key types (e.g., different data formats, identifiers) and standardize them for joining operations. This universal interface provides multi-functionality by handling various key formats through a single standardized mechanism, reducing the complexity of key mapping while maintaining the ability to join disparate databases.
4Adaptability or versatility
If datasets with different key types are joined, then the ability to combine knowledge from disparate databases is improved, but the difficulty of correlating data entries increases
Solution Approach 1:
The patent employs an intermediary key translation mechanism that automatically correlates data entries from datasets with different key types. This intermediary component detects key type differences, applies appropriate mapping rules, and produces correlated results without requiring manual intervention, thereby reducing correlation difficulty while maintaining data integration capability.
Solution Approach 2:
The patent implements self-service key correlation where the system automatically detects key type mismatches and performs the necessary transformations using predefined mapping rules. This self-service mechanism eliminates the need for manual correlation efforts, reducing difficulty while preserving the ability to integrate and correlate data from disparate databases with different key structures.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method of joining a first dataset configured to store a set of data entries each identified by a respective key of a first type and a second dataset configured to store a second set of data entries identified by a respective key of a second type, the method comprising: selecting an intermediate mapping entity from a set of possible intermediate mapping entities, each mapping entity storing association between keys of the first type and keys of the second type; providing the selected intermediate mapping entity for use in joining the first data set with the second data set; wherein the step of selecting the intermediate mapping entity is based on the intersection weight between the first and second data sets via each of the intermediate mapping entities, wherein the intersection weight is the proportion of overlapping data entries between the first and second datasets.