Hybrid Database Stream Processing Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stream processing engines and database systems face challenges in optimizing continuous queries due to increasing complexity and heterogeneity, leading to suboptimal throughput and resource utilization, especially when handling high-velocity sensor data.
Innovation Solution
A cost-based optimizer is designed for hybrid database and stream processing systems to determine optimal placement of operators across stream processing engines and database engines, using a two-phase optimization strategy to maximize throughput and minimize resource utilization by exploiting distributed execution and pipelined execution models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If continuous queries are executed using traditional stream processing engines or database systems, then the system can handle event stream processing, but the throughput is suboptimal due to increasing query complexity and heterogeneity
Solution Approach 1:
The patent segments the continuous query execution into multiple operators that can be independently optimized and distributed across different execution engines. Each operator in the query plan can be evaluated separately, allowing the system to handle complex queries by breaking them down into manageable segments that can be processed in parallel or distributed across hybrid cloud and on-premises infrastructure.
Solution Approach 2:
The patent introduces a new dimension of execution by implementing a hybrid architecture that combines stream processing engines and database systems. This dimensional expansion allows queries to be executed across multiple platforms simultaneously, with the optimizer selecting the most appropriate execution engine for each operator based on cost models, thereby increasing throughput without proportionally increasing complexity.
2Productivity
If the system uses a unified execution approach for continuous queries, then implementation is simpler, but resource utilization is suboptimal
Solution Approach 1:
The patent implements dynamic operator placement where the execution engine for each query operator is determined at optimization time based on cost models and can change based on workload characteristics. The system dynamically selects whether to execute operators on stream processing engines or database systems, allowing resource utilization to adapt to varying query patterns and data characteristics, thereby improving efficiency without requiring manual configuration.
Solution Approach 2:
The patent changes the parameter of execution engine selection from a static unified approach to a dynamic multi-engine approach. The cost-based optimizer evaluates multiple execution plans with different engine assignments and selects the optimal plan based on estimated costs, allowing the system to maximize resource utilization by matching query operators with the most suitable execution engines for their specific requirements.
3Productivity
If operator placement is determined without optimization, then deployment is faster, but throughput is reduced
Solution Approach 1:
The patent performs preliminary optimization by pre-calculating the optimal execution plan for continuous queries before they are executed. The cost-based optimizer evaluates multiple execution plans and determines the best operator placement strategy in advance, storing this optimized plan for reuse. This preliminary action allows the system to achieve high throughput during execution without incurring optimization overhead for each query run, as the optimization is performed once during plan generation.
Data Source
AI summary
The present disclosure involves systems, software, and computer implemented methods for optimizing continuous queries for hybrid execution over a stream processing engine and an in-memory database. In one example, a method may include identifying a continuous query executed at a stream processing engine, the continuous query including a plurality of operators. An optimal plan for execution of the identified continuous query at the stream processing engine is determined. For each of the plurality of operators in the determined optimal plan, an optimized placement decision for executing a particular operator in the stream processing engine or at a database system is determined. An optimized continuous query is generated from the identified continuous query based on the determined optimal placement decisions for each of the plurality of operators in the determined optimal plan. The optimized continuous query is then executed at the stream processing engine and the database system.