Message Stream Processor Microbatching for Router Failure Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inter-stream messaging systems face challenges in efficiently recovering from router failures while preserving message ordering, as existing methods require significant storage and processing resources, leading to inefficiencies and potential message loss.
Innovation Solution
The implementation of heartbeat messages in input and output streams acts as watermarks for recovery, allowing for accurate identification of processed messages and efficient restart of message routing without the need for extensive querying, using a small data structure to track target heartbeat messages and configurable insertion frequencies to balance data volume and processing power.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing recovery methods are used to preserve message ordering during router failure, then message ordering is maintained, but storage and processing resources are significantly consumed
Solution Approach 1:
The patent applies preliminary action by inserting heartbeat messages into input streams before failure occurs. These heartbeat messages serve as pre-positioned markers that enable rapid recovery without requiring extensive querying of message queues. The heartbeat messages are inserted at configurable intervals, creating a ready-to-use recovery mechanism that reduces both storage requirements and processing overhead during failure recovery operations.
2Measurement precision
If heartbeat messages are inserted frequently to improve recovery accuracy, then recovery precision increases, but data volume and processing power requirements increase
Solution Approach 1:
The patent applies parameter changes by making the heartbeat message insertion frequency configurable. This allows system operators to adjust the frequency parameter based on specific recovery accuracy requirements and available processing resources. The configurable frequency enables optimization of the balance between recovery precision and data volume, allowing the system to adapt to different operational conditions and resource constraints.
Data Source
AI summary
Embodiments provide a batching system that conforms message batches to publication constraints and also to message ordering requirements. An output array of messages is formed from messages received from a plurality of input streams, in which the messages are ordered. The output array preserves the ordering of the messages found in the source input streams. Messages are added from a head of the output array to a batch until addition of a next message to the batch would violate a particular batch processing constraint imposed on the batch. According to embodiments, one or more additional messages are included in the current batch when addition of the one or more additional messages to the batch (a) does not violate the particular batch processing constraint, and (b) continues to preserve the ordering of the messages, in the batch, with respect to the respective ordering of each of the plurality of input streams.