Micro-batch Snapshot Management for Stateful Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database systems are inadequate for processing continuous data streams, as they are designed for finite data sets and lack the processing flexibility required for modern event-driven applications such as sensor data, financial tickers, and network monitoring, which necessitate a different querying mechanism capable of handling long-running queries over unbounded data sets.
Innovation Solution
A method and system for processing events in event streams using a Continuous Query Language (CQL) engine, which includes receiving and processing micro-batch streams of input events, generating snapshots of the system's current state, and managing these snapshots using a snapshot management algorithm to support stateful processing and pattern matching in micro-batching stream processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional database systems are used for processing data, then data can be stored and queried using standard SQL, but the system cannot handle continuous data streams or perform long-running queries over unbounded data sets
Solution Approach 1:
The system segments continuous data streams into discrete micro-batches for processing. Each micro-batch is handled as a separate unit by the CQL engine, allowing traditional batch processing techniques to be applied to streaming data while maintaining continuity through iterative processing of successive batches
Solution Approach 2:
The system implements dynamic query execution where SQL queries are continuously re-executed on new micro-batches of data. The query processing is adaptive, automatically adjusting to incoming data streams and maintaining long-running query states without requiring fundamental changes to the database architecture
2Duration of action of moving object
If SQL queries are executed on finite data sets, then queries produce static results, but the system cannot perform long-running queries over continuous unbounded data sets
Solution Approach 1:
The system performs preliminary actions by establishing query execution contexts and state before data arrives. Snapshots of query state are captured and preserved, allowing queries to be resumed and continued across multiple micro-batch processing cycles, enabling long-running queries to maintain their execution context over extended periods
Solution Approach 2:
The system creates copies of query state through snapshots that are stored and preserved across micro-batch boundaries. These snapshot copies allow the query execution to be restarted and continued from previous states, ensuring reliability and accuracy of long-running queries over continuous data streams
3Adaptability or versatility
If snapshots are generated for state management in micro-batch processing, then stateful processing and pattern matching are enabled, but additional complexity is introduced in managing snapshot directories and lists
Solution Approach 1:
The system introduces an intermediary snapshot management layer that mediates between the micro-batch processing engine and the underlying storage system. This intermediary handles snapshot creation, storage, retrieval, and cleanup operations, providing a simplified interface while managing the complexity of snapshot directory structures and lifecycle management internally
Data Source
Figure `1
Figure 2
Figure 3
AI summary
A method for processing of a micro-batching stream to support fully stateful query processing is disclosed. The method comprises: receiving, by a computing device, a continuous query; applying, by the computing device, a transformation to the continuous query to generate a query plan for the continuous query; monitoring, by the computing device, the continuous query using a monitoring transformation process; receiving, by the computing device, a micro-batch stream of input events related to an application; and processing, by the computing device, the input events of the micro-batch stream based at least in part on the monitoring transformation process to generate a set of output events related to the application.