Fault Tolerant Batch Processing via Selective Checkpointing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems lack effective fault tolerance mechanisms for batch data processing, leading to repeated costly and complex steps during recovery from faults, especially when communicating with remote servers or performing high-cost actions.
Innovation Solution
Implementing a checkpoint process in a dataflow graph that saves intermediate results in non-volatile memory, allowing for reuse in case of faults, thereby reducing the need for repeated actions and minimizing the complexity and expense of checkpointing the entire system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If checkpointing is implemented for the entire system, then fault tolerance is improved, but device complexity increases
Solution Approach 1:
The patent divides the dataflow graph into multiple components and selectively applies checkpointing only to specific components that perform costly actions (such as remote server communications), rather than implementing checkpointing across the entire system. This segmentation allows fault tolerance to be improved for critical operations while avoiding the complexity overhead of system-wide checkpointing.
Solution Approach 2:
The patent applies different quality levels of fault tolerance to different parts of the system based on their specific needs. Components that perform expensive operations (e.g., remote server calls) are equipped with checkpointing capabilities, while other components operate without checkpointing. This local quality approach optimizes the balance between fault tolerance and complexity by tailoring the level of protection to each component's requirements.
2Stability of the object's composition
If all components are reprocessed after a fault, then data consistency is improved, but loss of time increases
Solution Approach 1:
The patent performs preliminary actions by saving intermediate results from costly operations (such as remote server communications) into checkpoint buffers during normal processing. When a fault occurs, these pre-saved results can be retrieved and reused, eliminating the need to reperform expensive operations and significantly reducing reprocessing time while maintaining data consistency.
Solution Approach 2:
The patent implements a mechanism to discard the need for reprocessing by recovering previously computed results from checkpoint buffers. Instead of reprocessing all data after a fault, the system recovers intermediate results from storage and continues processing from that point, thereby maintaining data consistency while minimizing time loss.
3Reliability
If costly actions are repeated during fault recovery, then reliability is improved, but loss of energy increases
Solution Approach 1:
The patent creates copies of intermediate results from costly operations and stores them in checkpoint buffers. During fault recovery, instead of reperforming energy-intensive operations, the system retrieves and uses these pre-computed copies, thereby maintaining reliable fault recovery while significantly reducing energy consumption.
Data Source
AI summary
Among other aspects disclosed are a method and system for processing a batch of input data in a fault tolerant manner. The method includes reading a batch of input data including a plurality of records from one or more data sources and passing the batch through a dataflow graph. The dataflow graph includes two or more nodes representing components connected by links representing flows of data between the components. At least one but fewer than all of the components includes a checkpoint process for an action performed for each of multiple units of work associated with one or more of the records. The checkpoint process includes opening a checkpoint buffer stored in non-volatile memory at the start of processing for the batch.