Counting Bloom Filter Duplicate Removal for Aggregate Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face inefficiencies in processing Multiple Distinct Aggregate (MDA) type queries due to the need for replicating data streams and extensive duplicate elimination processes, which consume significant resources in terms of CPU, I/O, and memory usage.
Innovation Solution
The implementation of Counting Bloom Filters and a DISTINCT HASH operator to identify and remove duplicates within the data stream during the 'group by' operation, allowing for early duplicate removal and minimizing the need for replicating the input data stream, thereby reducing processing overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data stream replication is performed to handle multiple distinct aggregate queries, then query processing capability is improved, but CPU usage, I/O usage, and memory usage increase significantly
Solution Approach 1:
The patent combines multiple distinct aggregate operations into a single unified processing path by using a hash operator that can handle multiple distinct columns simultaneously. Instead of replicating the data stream for each distinct aggregate operation, the system merges them into one stream and processes all distinct aggregates in parallel within the same execution plan, thereby reducing CPU usage while maintaining query processing capability.
Solution Approach 2:
The patent segments the distinct aggregate operations by column, allowing each distinct column to be processed independently through the hash operator. This segmentation enables the system to avoid full data stream replication while still handling multiple distinct aggregates efficiently, as each column's distinct values can be computed separately from the unified data stream.
2Productivity
If data stream replication is performed to handle multiple distinct aggregate queries, then query processing capability is improved, but I/O usage increases significantly
Solution Approach 1:
The patent merges multiple distinct aggregate operations into a single I/O path by processing all distinct columns from one unified data stream. This eliminates the need for separate I/O operations for each distinct aggregate, thereby reducing I/O usage while maintaining the ability to process multiple distinct aggregates simultaneously.
3Productivity
If data stream replication is performed to handle multiple distinct aggregate queries, then query processing capability is improved, but memory usage increases significantly
Solution Approach 1:
The patent merges multiple distinct aggregate operations into a single memory structure by using a shared hash operator that processes all distinct columns from one data stream. This eliminates the need for separate memory allocations for each distinct aggregate operation, thereby reducing memory usage while maintaining query processing capability.
4Measurement precision
If extensive duplicate elimination processes are performed on replicated data streams, then distinct aggregate accuracy is improved, but computational time increases
Solution Approach 1:
The patent performs preliminary duplicate elimination by using a hash operator to compute distinct values for each column before the final aggregation step. This preliminary action ensures that duplicates are removed early in the processing pipeline, maintaining distinct aggregate accuracy while reducing the computational time required for subsequent aggregation operations.
Solution Approach 2:
The patent segments the duplicate elimination process by column, allowing the hash operator to process each distinct column independently and in parallel. This segmentation maintains accuracy for each distinct aggregate while reducing overall computational time through parallel processing of multiple columns simultaneously.
Data Source
AI summary
There is disclosed a system and method for executing multiple distinct aggregate queries. In an embodiment, the method comprises: providing at least one Counting Bloom Filter for each distinct column of an input data stream; reviewing count values in the at least one Counting Bloom Filter for the existence of duplicates in each distinct column; and if necessary, using a distinct hash operator to remove duplicates from each distinct column of the input data stream, thereby removing the need for replicating the input data stream and minimizing distinct hash operator processing. Also, the use of Counting Bloom Filters for monitoring data streams allow an early duplicate removal of the input stream of data, resulting in savings in computation time and memory resources.