Query Aware Partitioning for High-Rate Data Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Data Stream Management Systems (DSMS) face challenges in processing high-rate data streams due to the computational limitations of single centralized servers, which struggle to keep up with the demands of applications generating data at speeds exceeding 100 million packets per second, leading to inefficient query processing.
Innovation Solution
A method and system for query-aware partitioning, which involves classifying queries, computing optimal partition sets, reconciling them for compatibility, and selecting the most efficient partition sets to distribute data across multiple nodes, optimizing query plans to maximize local data processing and reduce network load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single centralized server is used for data stream processing, then system simplicity is maintained, but processing capability becomes insufficient for high-rate data streams exceeding 100 million packets per second
Solution Approach 1:
The patent applies segmentation by dividing the centralized server into multiple distributed nodes that process data streams independently. Each node handles a portion of the data load, enabling the system to process high-rate data streams exceeding 100 million packets per second while maintaining scalability through additional nodes as needed.
2Productivity
If data is processed centrally, then coordination is simplified, but query processing efficiency decreases due to computational limitations
Solution Approach 1:
The patent segments queries into distributed execution plans where each query node processes locally available data independently. This reduces the computational burden on any single central processor and improves query processing efficiency by parallelizing operations across multiple nodes.
Solution Approach 2:
The patent introduces a coordinator as an intermediary component that manages query distribution and result aggregation across distributed nodes. The coordinator receives data from multiple sources, distributes appropriate queries to relevant nodes, and aggregates results, thereby simplifying coordination while maintaining high processing efficiency.
3Loss of energy
If more data is processed locally, then network load is reduced, but partitioning compatibility between queries becomes more difficult to maintain
Solution Approach 1:
The patent performs preliminary partitioning of data streams into compatible partition sets before query execution. By pre-computing partitioning schemes that satisfy compatibility requirements across multiple queries, the system reduces network load through local processing while avoiding complex runtime reconciliation of partition sets.
Data Source
AI summary
A method and system for providing query aware partitioning are disclosed. For example, the method receives a query plan comprising a plurality of queries, and classifies each one of the plurality of queries. The method computes an optimal partition set for each one of the plurality of queries, and reconciles the optimal partition set of each one of the plurality of queries with at least one subset of queries of the plurality of queries. The method selects at least one reconciled optimal partition set to be used by each query of the plurality of queries, and stores the selected at least one reconciled optimal partition set in a computer readable medium.