Stream Processing Anchor Recovery for Independent Node Startup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stream processing systems face challenges in efficiently and effectively starting up and recovering computing nodes, leading to inefficiencies in network bandwidth and computing resource consumption.
Innovation Solution
The use of anchors in stream processing systems, where each node determines its phase based on pending requests, generates or receives anchors for a minimum time, and stores them in a journal for recovery, allowing independent recovery without cascading failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If traditional stream processing startup and recovery methods are used, then system reliability is maintained, but network bandwidth and computing resource consumption increase
Solution Approach 1:
The patent applies preliminary action by having computing nodes proactively determine their phase based on pending requests before actual failures occur. Nodes pre-calculate recovery anchors and store them in journals, so when a failure happens, recovery can immediately use these pre-prepared anchors without cascading failures or excessive resource consumption during the recovery process
Solution Approach 2:
The patent segments the stream processing system into independent computing nodes that can recover individually. Each node maintains its own journal with anchors specific to its phase, allowing isolated recovery without affecting other nodes. This segmentation prevents cascading failures and reduces overall network bandwidth consumption during recovery operations
2Productivity
If traditional startup protocols are used, then all nodes can start processing, but startup efficiency decreases due to cascading failures
Solution Approach 1:
The patent implements feedback mechanisms where computing nodes continuously monitor and determine their phase based on pending requests from downstream nodes. This feedback loop allows nodes to dynamically adjust their anchor selection and recovery process, preventing cascading failures and enabling faster, more efficient startup by only recovering nodes that actually need recovery
Solution Approach 2:
Nodes perform preliminary phase determination and anchor selection before startup completion, so when startup is triggered, the system can immediately resume processing with minimal delay. This preliminary preparation eliminates the need for time-consuming cascading recovery sequences
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Described herein is a system and method for startup and/or recovery for stream processing. During a startup phase: start anchor request(s), each identifying a particular time, are accumulated until request(s) are pending from downstream nodes. A minimum time of the accumulated start anchor request(s) is determined. If the processing system is an input node, an anchor associated with the determined minimum time is generated. Otherwise, a start anchor request is provided to an upstream node identifying the determined minimum time. Once the anchor associated with the determined minimum time is received (or generated), the anchor is provided in response to a polled start anchor request anchor for the determined minimum from a downstream node. Asynchronous requests for batches of data bounded by two specific anchors are performed in accordance with information stored in an ordered collection of anchors during a recovery phase.