Distributed Streaming Nodes Local Failure Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for failure recovery in streaming data processing systems are inefficient, especially over wide-area-networks where centralized storage or message queueing systems are not readily available.
Innovation Solution
The system employs a distributed architecture where nodes independently manage failure recovery by transmitting data using a protocol that determines data ownership for recovery purposes, allowing for local and independent recovery without centralized checkpointing or logging.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If centralized checkpointing or logging systems are used for failure recovery, then recovery reliability is improved, but system complexity and overhead increase
Solution Approach 1:
The patent extracts the centralized checkpointing and logging functionality from the system, replacing it with a distributed approach where each node maintains its own state and failure information locally. This eliminates the need for complex centralized coordination while maintaining recovery capabilities through local state management and peer-to-peer communication.
Solution Approach 2:
Each node in the distributed system is equipped to manage its own failure recovery independently using its local state and the state of its peers. The system employs self-service mechanisms where nodes can autonomously detect failures, retrieve necessary state information from peers, and resume processing without requiring centralized control or complex coordination protocols.
2Reliability
If distributed checkpointing with coordination is implemented, then recovery consistency is improved, but runtime overhead increases
Solution Approach 1:
The patent segments the checkpointing process into independent local state snapshots maintained at each node, eliminating the need for global coordination. Each node maintains its own consistent state and failure information locally, allowing recovery to proceed independently at each node without requiring synchronized checkpoints across the entire distributed system.
Solution Approach 2:
Instead of requiring full system coordination for every recovery operation, the system performs partial actions locally at each node. Nodes maintain sufficient local state and peer state information to handle recovery independently, using only the minimal necessary coordination when needed, thereby reducing overall runtime overhead while maintaining consistency.
3Reliability
If entire system must rollback to common starting point for recovery, then recovery correctness is improved, but recovery latency increases
Solution Approach 1:
The patent segments the rollback operation into node-specific local rollbacks rather than a system-wide rollback. Each node independently rolls back to its own last known good state using its local state and failure information, eliminating the need to wait for the entire system to synchronize at a common checkpoint. This enables parallel recovery operations across multiple nodes, significantly reducing recovery latency while maintaining correctness through local state consistency.
Solution Approach 2:
The system performs preliminary actions by maintaining continuous local state snapshots and failure information at each node during normal operation. This preliminary state preservation enables nodes to immediately begin recovery from their own local states without requiring system-wide coordination or waiting for a common rollback point, thereby reducing recovery latency while ensuring correctness through the pre-maintained state consistency.
Data Source
AI summary
This patent document describes failure recovery technologies for the processing of streaming data, also referred to as pipelined data. The technologies described herein have particular applicability in distributed computing systems that are required to process streams of data and provide at-most-once and/or exactly-once service levels. In a preferred embodiment, a system comprises many nodes configured in a network topology, such as a hierarchical tree structure. Data is generated at leaf nodes. Intermediate nodes process the streaming data in a pipelined fashion, sending towards the root aggregated or otherwise combined data from the source data streams towards. To reduce overhead and provide locally handled failure recovery, system nodes transfer data using a protocol that controls which node owns the data for purposes of failure recovery as it moves through the network.