Distributed Database Join Using Active Flags and RDMA
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data mining solutions are inadequate for handling large datasets due to inefficiencies in joining database tables and lack of scalability, particularly when dealing with distributed databases and node failures.
Innovation Solution
The system employs active flag structures to identify table entries satisfying query criteria, uses direct memory access and point-to-point communications between nodes, and implements a fault-tolerant algorithm for distributing data across nodes, enabling efficient joins of distributed database tables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current join techniques are used for distributed database tables, then the system can handle basic queries, but the efficiency and scalability deteriorate when dealing with large datasets and multiple tables
Solution Approach 1:
The patent segments the join operation into multiple phases: (1) distributing data across nodes with hash functions, (2) performing local joins at each node, (3) collecting results. This segmentation allows parallel processing at multiple levels, dramatically improving efficiency while maintaining manageable complexity through modular design
Solution Approach 2:
The patent introduces a new dimension of parallelism by distributing the join operation across multiple computational nodes simultaneously. Instead of sequential processing, the join is performed in parallel across node boundaries, transforming a single-threaded operation into a multi-dimensional concurrent process that scales with hardware resources
2Adaptability or versatility
If data is distributed across multiple nodes for scalability, then the system can handle larger datasets, but the reliability deteriorates when nodes go offline
Solution Approach 1:
The patent implements fault tolerance by anticipating node failures beforehand. It uses redundant data distribution strategies where data is replicated across multiple nodes, and employs checkpointing mechanisms that save intermediate join results periodically. This cushioning ensures that if a node fails, the system can recover from saved checkpoints without complete failure
Solution Approach 2:
The patent applies local quality by allowing each node to independently handle its local data portion with specialized join algorithms optimized for local conditions. Each node maintains autonomy in processing its assigned data segment, enabling the system to continue operating with partial functionality even when some nodes fail, thus improving overall reliability through localized resilience
3Ease of manufacture
If traditional join algorithms are used, then the implementation is straightforward, but the processing time increases significantly for large distributed tables
Solution Approach 1:
The patent applies preliminary action by pre-distributing data across nodes using hash functions before the actual join operation begins. It also pre-computes join keys and prepares data structures in advance. This preliminary organization of data eliminates the need for complex runtime computations, significantly reducing actual join processing time while keeping the implementation relatively simple
Solution Approach 2:
The patent substitutes traditional mechanical join algorithms with optimized distributed algorithms that leverage network parallelism and local processing. Instead of moving all data to a single location for joining, the system replaces the centralized mechanical approach with a distributed computational model that processes joins locally and aggregates results, dramatically reducing communication overhead and processing time
Data Source
AI summary
Techniques to process a query and perform a join of tables that are distributed across nodes of a network. The join can be performed by analyzing a Where clause. An active flag structure can have flag values that identify table entries satisfying criteria of the Where clause. Keys of surviving entries of a first table can be used to generate a request for a second table to be joined. The request can be for second flags for the second table when the Where clause has criteria for the second table. A response can be used to update the first flags to change a first flag to False. After updating, data can be retrieved for first flags that are True. Requests can use identifiers associated with the first table that identify a location for sending the request, e.g., using RDMA or MPI.