Stream Computing Operator Shutdown Sequence for Data Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stream computing systems lack the ability to gracefully process already inputted data to completion during shutdown, leading to potential data loss, and existing methods for minimizing data loss are either ineffective or overly disruptive to system availability.
Innovation Solution
A method where a shutdown message is cascaded through operators in a stream computing application, causing source operators to stop data acquisition, send cached data to output queues, and then terminate, allowing processing operators to complete data processing and sink operators to write buffered outputs before terminating, ensuring data is processed to completion without loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a shutdown message is cascaded through operators to complete data processing, then data loss is prevented, but system downtime increases
Solution Approach 1:
The system performs preliminary actions by sending shutdown messages through the operator cascade before actual termination. Source operators send cached data to output queues, processing operators process data in their input queues, and sink operators write buffered outputs to destinations. This preliminary completion of data processing tasks before shutdown ensures no data is lost while the system transitions to a stopped state.
2Reliability
If data input is stopped prior to shutdown, then data loss is minimized, but already inputted data may not be processed to completion
Solution Approach 1:
The system maintains continuity of useful action by allowing processing operators to continue processing data that is already in their input queues even after source operators stop acquiring new data. The shutdown message cascades through the operators, ensuring that all data already in the system is processed to completion before operators terminate, thus maintaining productive action until the very end.
3Reliability
If a two-phase shutdown with delay is used, then data processing completion is attempted, but system availability is adversely impacted
Solution Approach 1:
The system uses dynamic operator behavior in response to shutdown messages. Instead of a static fixed delay, each operator dynamically adjusts its shutdown behavior: source operators stop data acquisition and send cached data, processing operators complete processing of data in their queues, and sink operators write buffered outputs. This dynamic response ensures data completion without requiring an arbitrary fixed delay that would unnecessarily impact availability.
Data Source
AI summary
In a stream computing application shutdown, a shutdown message is received by a source operator of the stream computing application. In response, the source operator stops acquiring data from external sources, sends any cached data to an output queue of the source operator, sends the shutdown message to the output queue of the source operator, and sends the cached data and shutdown message to an input queue of another operator in the stream computing application. The source operator then terminates. In response to receiving the shutdown message, the other operator completes the processing of data in its input queue and sends any outputs from the processing of the data in its input queue to one or more output destinations. The other operator then terminates. In this manner, a stream computing application may be shut down while ensuring that any already inputted data is processed to completion, thus avoiding data loss.