Stream Processing Fault Tolerance via Externalized State
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current stream processing systems face challenges in fault tolerance and data recovery, including modification of batch size during replay, over-counting when input is not a superset during replay, and inaccurate counter updates, especially when data is merged or mapped with external data stores.
Innovation Solution
The technology introduces a fault tolerance strategy that reruns all stages of a batch upon failure, restarts crashed worker nodes, swaps node-states for completed nodes, and maintains state for uncontactable nodes, while also addressing batch size modifications and counter updates by leveraging concurrent and multiplexed processing schemes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system reruns all stages of a batch upon failure to ensure data accuracy, then reliability is improved, but processing time increases
Solution Approach 1:
The system externalizes state data to a data store before batch processing begins, allowing worker nodes to reload their state quickly after failure without needing to recalculate from scratch. This preliminary state preservation enables faster recovery while maintaining processing accuracy.
Solution Approach 2:
The system creates copies of batch data and state information in external data stores, allowing failed worker nodes to retrieve their state from these copies rather than regenerating it. This copying mechanism enables rapid recovery while ensuring data consistency across reruns.
2Reliability
If the system restarts crashed worker nodes with fresh state to ensure clean processing, then reliability is improved, but resource wastage increases
Solution Approach 1:
Instead of creating entirely fresh state for restarted worker nodes, the system copies the previous valid state from external data stores to the restarted nodes. This selective copying preserves processed results and avoids redundant computation, reducing resource wastage while ensuring reliable error recovery.
3Reliability
If the system swaps node-states using externalized state data to recover from failures, then reliability is improved, but system complexity increases
Solution Approach 1:
The system introduces an external data store as an intermediary between worker nodes and their state data. This intermediary layer simplifies the complexity by providing standardized state persistence and retrieval operations, isolating the complexity of state management from the batch processing logic itself.
4Productivity
If the system maintains state for uncontactable worker nodes to enable continued processing, then productivity is improved, but measurement precision deteriorates due to inaccurate counter updates
Solution Approach 1:
The system implements feedback mechanisms where worker nodes periodically report their processing state and counter values to a central coordinator. This feedback loop allows the system to detect uncontactable nodes, maintain their state in external stores, and trigger appropriate recovery actions while ensuring counter accuracy through verification against the externalized state.
Data Source
AI summary
The technology disclosed relates to discovering multiple previously unknown and undetected technical problems in fault tolerance and data recovery mechanisms of modem stream processing systems. In addition, it relates to providing technical solutions to these previously unknown and undetected problems. In particular, the technology disclosed relates to discovering the problem of modification of batch size of a given batch during its replay after a processing failure. This problem results in over-count when the input during replay is not a superset of the input fed at the original play. Further, the technology disclosed discovers the problem of inaccurate counter updates in replay schemes of modem stream processing systems when one or more keys disappear between a batch's first play and its replay. This problem is exacerbated when data in batches is merged or mapped with data from an external data store.