Dynamic Continuous Query Deletion in Data Stream Management Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data stream management systems (DSMS) do not allow for dynamic deletion of continuous queries once they are registered and started processing event data, leading to inefficient resource usage as deleted queries continue to be executed, misleadingly appearing as removed.
Innovation Solution
Implementing a system that enables the dynamic deletion of continuous queries during normal operation by updating the query plan, releasing shared resources, and de-scheduling unused operators, allowing for real-time adjustment of query execution without interrupting the system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If a continuous query is deleted from the DSMS, then the system should release resources and reduce load, but the query continues to execute consuming resources
Solution Approach 1:
The system implements dynamic query deletion by allowing queries to be added and removed at runtime. The query plan is updated dynamically when deletion commands are received, enabling the system to adapt its execution plan without restarting or interrupting overall operations. This resolves the contradiction by making the query lifecycle flexible rather than fixed.
Solution Approach 2:
The system segments the query execution architecture into independent components (query registration, query execution, query deletion) that can be manipulated separately. By segmenting the query plan into individual query entries, the system can selectively remove specific queries while maintaining others, allowing precise resource management without affecting the entire system.
2Productivity
If the query plan is updated to remove a deleted query, then resource management improves, but system complexity increases
Solution Approach 1:
The system implements self-service through automatic query plan regeneration. When a query deletion command is received, the system automatically updates the query plan by removing references to the deleted query and regenerating execution plans for affected views. This automated self-update mechanism reduces manual intervention complexity while improving resource management efficiency.
Solution Approach 2:
The system uses feedback loops where deletion commands trigger automatic query plan updates. The system monitors query registration and deletion events, and automatically adjusts the execution plan based on these events. This feedback mechanism ensures the query plan remains synchronized with the actual set of active queries, improving resource efficiency without requiring complex manual management.
Data Source
AI summary
A first continuous query is deleted from several continuous queries that are being executed, by performing different actions that depend on how resources are shared therebetween. As an example, a second continuous query is deleted if a view is referenced by the first continuous query, and if the view is built on the second continuous query, and if the view is not referenced by any query other than the first continuous query. Such deletion is followed by compiling the second continuous query, followed by updating the view to refer to a newly-compiled version of the second continuous query. As another example, if an operator in the first continuous query is not used by any other query, this operator is de-scheduled from execution, and any exclusively-owned resources are released. Also, data to be read by the operator and data generated by the operator, if present in shared resources, are deleted therefrom.