Event Batching and Checkpoint Markers for Stream Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database systems are inadequate for processing continuous data streams, as they are designed for finite data sets and lack the processing flexibility required for modern applications that generate event streams, such as sensor data and financial tickers, which necessitate a new approach for managing and querying unbounded data sets.
Innovation Solution
An event processing system that processes continuous input streams, generates output streams, and employs checkpoint marker events to create event batches, allowing for efficient recovery from system failures by storing snapshots of the system state and using output sequencing to ensure precise recovery of events.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional database systems are used to process continuous data streams, then the system structure remains simple and familiar, but the system cannot handle unbounded data sets and lacks processing flexibility for event stream-based applications
Solution Approach 1:
The patent segments the continuous event stream into discrete batches using checkpoint markers. Each batch is processed independently, allowing the system to handle unbounded data streams by breaking them into manageable units. This segmentation enables event stream processing capability while maintaining a structured approach that doesn't overly complicate the system architecture.
Solution Approach 2:
The patent implements preliminary actions by inserting checkpoint markers into the event stream before processing. These markers pre-establish batch boundaries and enable the system to perform recovery operations more efficiently. The checkpoint markers are placed in advance, allowing the system to maintain simplicity while gaining the ability to handle continuous streams with controlled processing intervals.
2Reliability
If the system processes continuous event streams with full recovery capability, then system reliability improves, but the time required to recover from failures increases due to reprocessing all events
Solution Approach 1:
The patent performs preliminary actions by inserting checkpoint markers at predetermined intervals into the continuous event stream before processing. These markers establish predetermined batch boundaries that enable the system to identify exactly which events need reprocessing after a failure. This preliminary structuring allows the system to maintain high reliability through complete recovery capability while minimizing recovery time by avoiding reprocessing of events that occurred before the last checkpoint.
Solution Approach 2:
The patent segments the event stream into batches using checkpoint markers, allowing the system to recover only the specific batch containing the failure point rather than reprocessing all events. This segmentation maintains reliability by ensuring complete recovery of affected events while reducing recovery time by eliminating unnecessary reprocessing of previous batches.
3Loss of information
If the system transmits all output events after recovery, then data completeness is ensured, but duplicate events may be transmitted causing data inconsistency
Solution Approach 1:
The patent assigns output sequence numbers to events as a preliminary action before transmission. This sequencing allows the system to track which events have been successfully transmitted and in what order. After recovery, the system can compare sequence numbers to determine which events need retransmission, ensuring data completeness while preventing duplicates by only retransmitting events with sequence numbers greater than the last successfully transmitted event.
Solution Approach 2:
The patent implements feedback mechanisms by tracking output sequence numbers and using them to control retransmission behavior. The system receives feedback about which events have been successfully processed and transmitted, then uses this information to determine the exact set of events that need retransmission after failure recovery. This feedback loop ensures complete data transmission while maintaining accuracy by excluding events that were already successfully transmitted.
4Productivity
If the system processes events in real-time continuously, then processing speed is maximized, but the system lacks checkpoint capability for efficient failure recovery
Solution Approach 1:
The patent implements periodic action by inserting checkpoint markers into the continuous event stream at predetermined intervals. This periodic checkpointing allows the system to maintain continuous real-time processing of events while periodically establishing recovery points. The periodic nature of checkpoint insertion minimizes interference with the continuous processing flow, maintaining high throughput while enabling efficient failure recovery without requiring complex continuous checkpointing mechanisms.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
This disclosure relates to a system comprising: a memory storing a plurality of instructions; and one or more processors configured to access the memory and execute the plurality of instructions to at least: receive a continuous input stream of events related to an application; determine an interval for inserting a checkpoint marker event into the stream of events, wherein the size of the interval is determined based at least in part on a type of the application, a latency requirement of the application, and a frequency at which events of the input stream of events are received; process the continuous input stream of events to generate an output stream of events related to the application, the processing comprising inserting the checkpoint marker event into the continuous input stream to create an event batch, and the event batch including each event of the continuous input stream of events received during the determined interval; determine an output sequence number for an output event in the output stream of events; transmit the output event in the output stream of events; store the output sequence number of the output event; while the continuous input stream of events is being processed, receive an indication of failure of the system; determine a current output sequence number of a most recently transmitted output event in the output stream of events; determine a most recently processed event batch; in response to the indication of failure of the system, re-process the events in the most recently processed event batch and determine a set of one or more output events of the output stream to be transmitted based on the current output sequence number and the most recently processed event batch, the set of one or more output events to be transmitted comprising each of the re-processed events having an output sequence number greater than the output sequence number of the most recently processed event; and transmit the set of one or more output events related to the application.