Query Processing Distribution in Continuous Computation Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Continuous processing systems face challenges in efficiently distributing query processing across a cluster of servers for streaming data, as existing methods are inadequate for handling the complexities of streaming data queries.
Innovation Solution
A method is introduced where a continuous computation system creates an execution plan by analyzing the semantics and requirements of queries to determine how to distribute processing across a cluster of servers, including considering manual hints, and applies specific strategies for queries that can process messages independently, in groups, with aggregator functions, or containing subqueries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If query processing is distributed over a cluster of servers for streaming data, then processing capacity and scalability are improved, but system complexity and difficulty of managing different query types increase
Solution Approach 1:
The patent segments query processing into distinct categories (independent messages, partitioned groups, aggregators, subqueries) and assigns specialized processing logic to each category. This segmentation allows the system to handle complex streaming queries by breaking them down into manageable segments that can be processed independently on different servers, thereby improving productivity while managing system complexity through structured organization.
Solution Approach 2:
The system performs preliminary analysis of query semantics and requirements before executing queries on the cluster. By pre-classifying queries into specific types and determining their processing strategies in advance, the system prepares execution plans that simplify runtime processing. This preliminary action reduces the complexity of real-time decision-making and enables efficient distribution of processing tasks across servers.
2Productivity
If processing logic is duplicated on multiple servers and input messages are divided, then load balancing and resource utilization are improved, but coordination overhead and communication costs increase
Solution Approach 1:
The patent divides input messages into distinct segments or partitions that can be processed independently by different servers. Each server handles a specific partition of the data stream, eliminating the need for constant coordination between servers during processing. This segmentation approach enables effective load balancing while minimizing coordination overhead, as servers operate autonomously on their assigned partitions.
3Productivity
If queries are processed independently on different servers, then processing speed and throughput are improved, but accuracy of aggregated results may deteriorate
Solution Approach 1:
The patent introduces intermediary mechanisms that coordinate between independent server processors. These intermediaries ensure that results from multiple servers are correctly aggregated and combined, maintaining measurement precision while allowing parallel processing to proceed independently. The intermediary layer handles the coordination of aggregated results without slowing down the individual processing speed of each server.
Data Source
AI summary
A method, in a continuous computation system, for processing a set of registered queries over a cluster of servers includes creating an execution plan for processing a set of queries over a cluster of servers, where the continuous computation system creates the execution plan by analyzing the semantics and requirements of the queries to determine how to distribute processing across the cluster. Analysis of a query can include determining whether input messages for the query can be processed independent of each other, whether input messages for the query can be partitioned into groups that can be processed independent of each other, whether the query includes an aggregator function, and whether the query includes a subquery.