Inner Join Processing on Sharded Data Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Performing inner joins on very large and highly sharded data sets is resource-intensive and time-consuming, often requiring significant computational resources and can fail due to memory constraints, especially when dealing with arbitrarily large input tables with varying key values that do not fit in memory.
Innovation Solution
The method involves partitioning key-value pairs into blocks, replicating these blocks across computing devices, and distributing pairs with matching block-keys for processing, allowing for efficient computation of inner joins by breaking down the task into smaller jobs that can be handled by multiple computing devices, thereby reducing memory requirements and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional inner join methods are used on very large data sets, then processing time becomes excessively long (days or weeks), but the system can handle the complete data set
Solution Approach 1:
The patent segments the inner join operation by partitioning both input data sets into blocks and processing them in parallel across multiple computing devices. Each device handles a subset of blocks simultaneously, dividing the monumental task into manageable units that can be processed concurrently, thereby reducing total processing time from days/weeks to hours.
Solution Approach 2:
The patent introduces a block-based dimensional structure to the traditional row-by-row join process. By organizing data into blocks and creating a block-key mapping system, the invention adds a hierarchical dimension that enables parallel processing and significantly accelerates the join operation across distributed computing devices.
2Reliability
If traditional inner join methods are used on arbitrarily large input tables, then memory constraints cause failures, but the system attempts to process complete data sets
Solution Approach 1:
The patent segments large input tables into manageable blocks, where each block contains a subset of key-value pairs. This segmentation allows the system to process data in chunks that fit within memory constraints of individual computing devices, preventing memory overflow while maintaining the ability to handle arbitrarily large data sets through distributed processing.
Solution Approach 2:
The patent introduces a block-key mapping as an intermediary structure that bridges the input data blocks and the final join results. This intermediary layer enables efficient coordination and matching between blocks from different input tables without requiring the entire data set to reside in memory simultaneously, thus overcoming memory constraints.
3Quantity of substance
If data is partitioned into smaller blocks for processing, then memory requirements are reduced, but the complexity of partitioning and replicating blocks increases
Solution Approach 1:
The patent performs preliminary partitioning of input data blocks and pre-computation of block-key mappings before the actual join operation. This preliminary action organizes the data structure in advance, reducing the complexity during the join process itself and enabling systematic replication of blocks across computing devices without requiring complex real-time coordination.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In a distributed database, an inner join of very large data sets is performed by distributing smaller jobs to a plurality of computing devices. For each key, values in a first input are partitioned into a first set block set, and values in a second input are partitioned into a second block set. For each key, the first block set is replicated by a number of blocks in the second block set, and the second block set is replicated by a number of blocks in the first block set. Each replicated block is assigned a block-key including the key and additional information to identify the replicated block. Each pair of replicated blocks having matching block-keys are distributed to one of a plurality of computing devices. Results for the inner join are received from the plurality of computing devices.