Inter-Stream Router Checkpointing via Heartbeat Watermarks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inter-stream messaging systems face inefficiencies in recovering from router failures, particularly due to the need for significant storage and processing resources to maintain and query persisted message records, leading to costly and inefficient recovery processes.
Innovation Solution
The implementation of heartbeat messages in input and output streams serves as watermarks for recovery, allowing for efficient identification of processed messages and minimizing data storage requirements, enabling recovery without querying message records, and balancing data volume with processing power.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If message records are persisted for recovery purposes, then reliability of message recovery is improved, but storage requirements and processing resources increase significantly
Solution Approach 1:
The patent extracts only the essential recovery information (sequence numbers and offsets) from the complete message records, storing only what is necessary for recovery while discarding the actual message content. This allows recovery reliability to be maintained while dramatically reducing storage requirements.
Solution Approach 2:
Instead of storing messages and recovering them after failure, the patent inverts the approach by storing only the position information (sequence numbers and offsets) that enable recovery without storing the actual message data. This inversion resolves the contradiction between reliability and storage.
2Reliability
If persisted message records are queried for recovery, then complete message recovery is achieved, but processing power and time consumption increase
Solution Approach 1:
The patent extracts only the critical recovery metadata (sequence numbers and consumer group offsets) from complete message records. This extraction enables rapid recovery by avoiding the need to query and process entire message records, thus reducing recovery time while maintaining completeness.
Solution Approach 2:
The patent performs preliminary action by pre-calculating and storing the sequence numbers and offsets before any failure occurs. This preliminary preparation enables instant recovery without requiring time-consuming queries of message records during the recovery process.
3Reliability
If all routed messages are tracked for recovery, then no message loss occurs, but device complexity and maintenance costs increase
Solution Approach 1:
The patent extracts only the essential tracking information (sequence numbers and consumer group offsets) from complete message state information. This reduction in tracked data maintains message tracking accuracy while significantly reducing system complexity and maintenance requirements.
Solution Approach 2:
The patent applies local quality by storing different types of information at different locations: sequence numbers are stored with input messages, while consumer group offsets are stored with output messages. This localized storage strategy maintains tracking accuracy while reducing overall system complexity.
Data Source
AI summary
Embodiments facilitate efficient recovery from an inter-stream messaging system failure by using heartbeat messages (HMs) to act as watermarks for message recovery. Embodiments insert HMs into each of the input streams at configurable regular intervals. The inter-stream router determines that a message being routed is a HM, and corresponding output HMs are generated based on the input HM. Embodiments insert a respective output HM into each of the output streams. Information indicating which output HMs have been processed from the output streams is tracked. After a failure of the inter-stream router, embodiments identify a target HM for each input stream, which is the latest HM sourced from the respective input stream that was processed from all of the output streams. After the inter-stream router restarts, the router initiates message routing, from each input stream, at the location of the respective input stream's target HM within the input stream.