Global ID Assignment for Distributed Equi-Join Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed databases, equi-join operations across split partitions face challenges due to the lack of correlation between locally generated value IDs, leading to meaningless IDs across partitions, which hinders efficient data communication and join operations.
Innovation Solution
Assigning global IDs to actual values that occur in multiple partitions, creating globalized lists in each partition to identify matching records, and using these lists for the join operation, ensuring that the same global ID is associated with an actual value across partitions, thereby enabling a consistent join condition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If each partition independently generates value IDs for local dictionaries, then storage efficiency is improved by using compact ID representations, but the value IDs become meaningless across different partitions, preventing efficient equi-join operations
Solution Approach 1:
The patent introduces a central dictionary as an intermediary component that maintains the authoritative mapping between actual values and global value IDs. Each partition's local dictionary serves as a cache referencing this central dictionary, ensuring that value IDs remain meaningful across partitions while still enabling efficient local operations. The central dictionary acts as the mediator that reconciles the need for compact storage with the requirement for cross-partition identifier consistency.
Solution Approach 2:
The patent merges the functionality of multiple independent local dictionaries into a unified global dictionary system. While each partition maintains its own local dictionary for efficient access, these local dictionaries are synchronized with and reference a central global dictionary that consolidates all value ID mappings. This merging approach allows partitions to benefit from local optimization while ensuring global consistency through the unified dictionary structure.
2Loss of substance
If actual values are replaced by value IDs in data tables, then data communication overhead is reduced, but the lack of global ID correlation across partitions makes equi-join operations difficult
Solution Approach 1:
The patent applies preliminary action by pre-establishing the global value ID mappings in the central dictionary before partitioned operations begin. Each partition pre-loads its local dictionary with mappings from the central dictionary, so that when equi-join operations are performed, the value ID correlations are already in place. This preliminary setup eliminates the need for complex runtime coordination between partitions during join operations.
Solution Approach 2:
The central dictionary serves as an intermediary that provides global value ID correlation information to all partitions. Instead of requiring direct communication and coordination between partitions during join operations, each partition can independently use its local dictionary (which references the central dictionary) to resolve value ID mappings. This intermediary approach simplifies the join operation complexity while maintaining reduced data communication overhead.
3Productivity
If a central dictionary is used to maintain global value ID mappings, then equi-join operations become feasible across partitions, but the system complexity increases due to the need for synchronization and maintenance
Solution Approach 1:
The patent segments the dictionary system into a central global dictionary and multiple distributed local dictionaries. The central dictionary maintains the authoritative global mappings, while each partition maintains a local dictionary that segments the global data into partition-specific portions. This segmentation allows the system to achieve global consistency through the central dictionary while reducing synchronization complexity by allowing local dictionaries to operate independently with periodic updates from the central source.
Data Source
AI summary
A join operation between split data tables includes providing reduction data from first partitions to each partition among second partitions. The reduction data serves to identify actual values in one of the second partitions that also occur in one of the first partitions. Global IDs are assigned. Translation lists including the global IDs are sent to the first partitions. Each first partition and each second partition create globalized lists which can then be combined to generate respective first and second compiled lists. The join operation can then be conducted on the first and second compiled lists.