Escrow Buffer Partitioning for Fault-Tolerant Data Replay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face challenges in maintaining fault tolerance and preventing data loss when components, such as the DoAll component in a dataflow graph, fail.
Innovation Solution
An escrow scheme is implemented, where processed data is stored in an escrow buffer associated with the ForAll procedure. This allows for data replay in case of component failure without reprocessing, and over-partitioning of data reduces the size of the escrow buffer for large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If processed data is stored in an escrow buffer to enable replay in case of component failure, then fault tolerance is improved, but the buffer size becomes very large for large collections of data
Solution Approach 1:
The patent divides the large collection of data into multiple partitions, and further divides each partition into smaller sub-partitions. This segmentation allows the escrow buffer to store only a subset of partition results at any given time, reducing the peak buffer size requirement while maintaining fault tolerance through the ability to replay results if needed
2Stability of the object's composition
If all data elements are processed and stored in escrow before release to consumer, then data consistency is improved, but the processing time increases due to waiting for persistent storage confirmation
Solution Approach 1:
The patent stores partition results in the escrow buffer before the consumer has persistently stored them, preparing the data in advance for potential replay. This preliminary action ensures that if a failure occurs, the results are already available in escrow without requiring reprocessing, thus maintaining consistency while reducing the time penalty to only the buffer storage duration rather than full reprocessing time
Data Source
AI summary
A method for fault-tolerant processing of a number of data elements using a distributed computing cluster. The distributed computing cluster includes a number of data processors associated with a corresponding number of data stores. The method includes storing the data elements in the distributed computing cluster, wherein the data elements are distributed across the data stores according to a number of partitions of data elements, processing data elements of a first set of partitions stored at a first data store using a first data processor to generate first result data for the data elements of the first set of partitions, sending the first result data from the distributed computing cluster to a consumer of the first result data outside the distributed computing cluster, and storing the first result data in a first buffer located in the distributed computing cluster and associated with the first data processor until the consumer has persistently stored the first result data outside the distributed computing cluster.