Storage-Side Filtering for Distributed Hash Joins
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed database systems are costly and complex to deploy and maintain, particularly in 'shared nothing' and 'shared disk' models, due to the need for extensive glue logic and cache coherency management, which can lead to inefficient data transfer and computational resource usage during hash join operations.
Innovation Solution
Implement storage-side filtering at the storage nodes to reduce data transfer by using bloom filters, allowing only filtered data that satisfies the join predicate to be sent to the query engine, thereby reducing communication bandwidth and computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed database systems use shared nothing or shared disk models, then data storage capacity and access performance are improved, but system complexity and deployment cost increase due to extensive glue logic and cache coherency management
Solution Approach 1:
The patent extracts the hash join operation from the query engine and moves it to the storage layer. This separation allows the storage layer to independently handle filtering operations using bloom filters, reducing the complexity of glue logic needed for coordinating join operations across distributed nodes while maintaining high data access performance.
Solution Approach 2:
The patent introduces bloom filters as an intermediary data structure at the storage layer. These filters act as a mediator between the query engine and stored data, enabling efficient pre-filtering of join candidates before data transfer, thus reducing cache coherency management requirements while preserving access performance.
2Reliability
If hash join operations are performed at the query engine, then join accuracy is ensured, but data transfer volume and network bandwidth consumption increase
Solution Approach 1:
The patent performs preliminary filtering of join candidates at the storage layer using bloom filters before data is transferred to the query engine. This preliminary action eliminates non-matching records in advance, ensuring that only potential join candidates are transferred over the network, thus maintaining join accuracy while significantly reducing bandwidth consumption.
Solution Approach 2:
The patent applies partial filtering at the storage layer using bloom filters, which provide probabilistic rather than deterministic matching. This partial action reduces data transfer volume by filtering out clearly non-matching records, while the query engine performs complete verification on the reduced dataset, balancing accuracy requirements with bandwidth efficiency.
3Measurement precision
If all data is transferred to the query engine for processing, then complete query accuracy is achieved, but computational resource usage and processing time increase
Solution Approach 1:
The patent segments the query processing workflow into two distinct phases: a filtering phase at the storage layer using bloom filters, and a verification phase at the query engine. This segmentation allows computationally intensive filtering operations to be distributed across storage nodes in parallel, reducing central processing time while maintaining overall query accuracy through subsequent verification.
Solution Approach 2:
The patent performs preliminary filtering of join candidates at the storage layer before data reaches the query engine. By pre-processing and eliminating non-matching records at the storage layer, the query engine receives a significantly reduced dataset requiring less computational time for verification, thus reducing total processing time while preserving query accuracy.
Data Source
AI summary
Storage-side filtering may be implemented for performing hash join operations with respect to data stored in distributed data storage. When a query to a database stored in a distributed data store is received, a hash join operation may be identified for performing the query. As part of performing the hash join operation, a query engine may cause storage nodes in the distributed data storing data for the database to filter data before sending the data to the query engine according to a join predicate for the hash join operation. A result of the query may then be provided to a user, using the filtered data provided from the storage nodes.