Database Semi-Join Filter Segmentation for Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed database systems, semi-join operations across multiple database partitions are resource-intensive due to the need for extensive data communication and memory usage, particularly when using bit filters for reducing data movement during join operations.
Innovation Solution
A data processing system that generates a reduced representation of rows from a source table and filters them using a target table's representation, identifying rows that do not need to be joined, thereby minimizing data transmission and memory requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If bit filters are used to reduce data movement during semi-join operations, then communication overhead is reduced, but memory usage increases significantly
Solution Approach 1:
The patent divides the bit filter into multiple segments or chunks that can be processed and transmitted in smaller units. Instead of broadcasting the entire bit filter to all database partitions, the system segments the filter data and processes it in distributed units, reducing the memory burden on each partition while maintaining the communication reduction benefits.
Solution Approach 2:
The system performs preliminary filtering operations using the bit filter before actual data transmission. By pre-processing the filter data and identifying matching rows in advance, the system reduces the volume of data that needs to be communicated between partitions, thereby reducing communication overhead without requiring all partitions to hold the complete bit filter in memory.
2Productivity
If bit filters are broadcast to all database partitions, then join operation efficiency improves, but communication resources are consumed
Solution Approach 1:
Instead of uniformly broadcasting bit filters to all database partitions, the patent applies local quality by sending bit filters only to the specific partitions that need them for join operations. Each partition receives filtering information tailored to its local data characteristics, improving join efficiency for relevant partitions while avoiding unnecessary communication with partitions that don't require the filter.
Solution Approach 2:
The system performs preliminary analysis to determine which database partitions require bit filters before initiating broadcast. By pre-identifying the target partitions that will benefit from the filtering operation, the system sends bit filters only to those specific partitions, thereby maintaining join operation efficiency while minimizing communication resource consumption across the network.
3Reliability
If complete data is transmitted between database partitions, then join accuracy is maintained, but data transmission volume increases
Solution Approach 1:
The patent extracts only the essential filtering information from the complete data set and transmits this extracted subset between database partitions. By taking out the critical join attributes and bit filter information while leaving out redundant data, the system maintains join accuracy for the extracted fields while dramatically reducing the overall data transmission volume between partitions.
Solution Approach 2:
Instead of transmitting complete data records, the system creates and transmits simplified copies containing only the necessary join attributes and bit filter information. These copied representations preserve the essential information needed for accurate join operations while occupying minimal transmission bandwidth, thereby maintaining join accuracy with reduced data transmission volume.
Data Source
AI summary
Disclosed is a data processing system, and an article of manufacturing for use with the data processing system. The data processing system joins rows associated with a column of a source table with rows associated with a column of a target table. The data processing system includes a source node containing the source table and including a target node containing the target table. The data processing system further includes a generating module for generating a reduced representation of selected rows associated with the column of the source table, and generating a representation of the column of the target table, a filtering module for filtering the generated reduced representation of selected rows associated with the column of the source table through the generated representation of the column of the target table, the filtered generated reduced representation of selected rows identifying source table rows that do not have to be joined with the target table, and a joining module for joining, to the rows associated with the column of the target table, the rows associated with the column of the source table minus the filtered generated reduced representation of selected rows.