Asynchronous Checkpointing for Distributed Graph Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current fault tolerance approaches in distributed graph processing networks are slow and do not allow for asynchronous processing, leading to inefficiencies in recovery from device failures and peak bandwidth utilization during checkpointing.
Innovation Solution
Implementing globally inconsistent checkpointing and asynchronous minimum recovery, which allows for faster recovery from device failures, reduces peak bandwidth utilization, and enables fine-tuning of checkpointing frequency based on device reliability and features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If classic Chandy-Lamport snapshot algorithm is used for fault tolerance, then reliability is improved, but processing speed deteriorates and asynchronous processing is not allowed
Solution Approach 1:
The patent segments the checkpointing process into device-level independent operations rather than requiring global synchronization. Each device performs checkpointing independently at its own pace, eliminating the need for coordinated stopping across all devices. This segmentation allows asynchronous processing while maintaining fault tolerance capabilities.
Solution Approach 2:
The patent introduces dynamic checkpointing where devices can perform checkpoints at different times based on their individual needs and progress. The system adapts to varying device speeds and workloads, allowing faster devices to proceed without waiting for slower ones, thus improving overall processing speed while maintaining reliability through selective checkpointing.
2Stability of the object's composition
If synchronous checkpointing is performed across all devices, then consistency is improved, but peak bandwidth utilization increases
Solution Approach 1:
The patent divides the checkpointing operation into independent device-level segments that can execute concurrently rather than requiring a unified global checkpoint. This segmentation reduces the simultaneous data transmission requirements, lowering peak bandwidth utilization while maintaining sufficient consistency for fault recovery through local checkpoint validity.
Solution Approach 2:
The patent implements partial checkpointing where not all devices need to participate in every checkpoint operation. Devices can perform checkpoints independently when needed, rather than requiring complete system-wide synchronization. This partial action approach reduces the total data volume transmitted at any given time, decreasing peak bandwidth requirements.
3Reliability
If all devices roll back to last snapshots during failure recovery, then reliability is improved, but recovery time increases
Solution Approach 1:
The patent extracts the rollback requirement from all devices and applies it only to the subset of devices that actually failed or are affected by the failure. Healthy devices that have valid checkpoints can continue processing without rolling back, significantly reducing the total recovery time while maintaining recovery correctness for the affected portion of the system.
Solution Approach 2:
The patent introduces dynamic recovery strategies where the rollback scope is adjusted based on the specific failure scenario. Instead of a fixed all-or-nothing rollback approach, the system dynamically determines which devices need to rollback and which can continue, optimizing recovery time while ensuring reliability for the necessary subset of devices.
4Reliability
If checkpointing frequency is increased for heterogeneous devices, then reliability is improved, but processing efficiency deteriorates
Solution Approach 1:
The patent applies local quality by allowing each device in the heterogeneous network to have its own checkpointing frequency tailored to its specific reliability characteristics, processing speed, and workload. Faster or more reliable devices can checkpoint less frequently, while slower or less reliable devices checkpoint more often, optimizing the balance between reliability and efficiency for each local component rather than imposing a uniform global frequency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A first device comprises: a memory configured to store a first sub-graph that is part of a distributed graph associated with a distributed graph processing network; a processor coupled to the memory and configured to: process the first sub-graph; and save, independently of a second device in the distributed graph processing network, a first snapshot of a first execution state of the first device at a first iteration time; and a transmitter coupled to the processor and configured to transmit the first snapshot to the second device or to a third device.