Streaming Aggregation Queries for Memory Bottleneck Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed database systems face memory-related bottlenecks during high-cardinality aggregation queries, leading to inefficient peak memory usage and potential crashes, especially when handling large data volumes.
Innovation Solution
The proposed solution involves partitioning group-by key values into disjointed and intersecting sets, where disjointed keys are streamed to reduce memory usage, and aggregation operations are performed on intersecting keys, allowing for efficient peak memory management without substantial CPU overhead, applicable to both low- and high-cardinality scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data entries are loaded into memory for aggregation processing, then aggregation accuracy is maintained, but peak memory usage increases causing bottlenecks and potential crashes
Solution Approach 1:
The patent segments data entries into two distinct subsets: those with disjointed keys and those with intersecting keys. This segmentation allows the system to process different data types differently - streaming disjointed keys to reduce memory usage while maintaining aggregation accuracy for intersecting keys that require full processing.
Solution Approach 2:
The patent extracts and identifies intersecting keys from the data stream, separating them from disjointed keys. By taking out only the necessary intersecting keys for memory-based aggregation processing while streaming the majority of disjointed keys, the system significantly reduces peak memory usage while preserving aggregation correctness.
2Quantity of substance
If data is streamed to reduce memory usage, then peak memory usage decreases, but processing complexity increases due to need to identify disjointed vs intersecting keys
Solution Approach 1:
The patent performs preliminary identification of intersecting keys before the main aggregation process. By pre-processing the data to determine which keys are intersecting versus disjointed, the system simplifies subsequent processing - disjointed keys can be directly streamed while intersecting keys are prepared for memory-based aggregation, reducing overall processing complexity.
3Ease of operation
If traditional aggregation processes all data in memory, then processing simplicity is maintained, but productivity decreases due to memory bottlenecks
Solution Approach 1:
The patent segments the aggregation process into two parallel pathways: streaming aggregation for disjointed keys and memory-based aggregation for intersecting keys. This segmentation enables the system to process large volumes of data efficiently through streaming while maintaining correctness for intersecting keys, thereby improving productivity without significantly increasing operational complexity.
Data Source
AI summary
An example computing device for streaming aggregation queries is provided. The computing device comprises a processor and memory storing instructions that cause the processor to receive a query from a caller, wherein the query comprises an aggregation operator, retrieve data comprising a plurality of data entries, determine first and second subsets of data entries from the plurality of data entries, wherein the first subset of data entries comprises data entries having disjointed keys and the second subset of data entries comprises data entries having intersecting keys, return the first subset of data entries to the caller, release the first subset of data entries from the memory, after releasing the first subset of data entries from the memory, aggregate the second subset of data entries using an aggregation operation corresponding to the aggregation operator, and return the aggregated second subset of data entries to the caller.