Distributed Query Pipelining with Bubble Fault Tolerance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face a trade-off between performance and fault-tolerance, as methods like DAG frameworks are either highly fault-tolerant but perform poorly due to sequential disk I/O, or highly performant but not fault-tolerant due to gang scheduling requirements that fail with limited resources.
Innovation Solution
Divide distributed computing job graphs into sub-graphs (bubbles) for independent execution, allowing intra-bubble communication via in-memory or network streaming and inter-bubble results to be stored in durable storage for fault-tolerance and performance optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If DAG frameworks are used with sequential disk I/O, then fault-tolerance is improved, but performance deteriorates
Solution Approach 1:
The job graph is segmented into multiple bubbles (sub-graphs) that can be executed independently. Each bubble represents a fault-tolerance boundary, allowing parallel execution of multiple bubbles while maintaining fault-tolerance within each segment. This segmentation enables concurrent processing of different job portions without requiring sequential disk I/O for every operation.
Solution Approach 2:
Durable storage acts as an intermediary between bubbles, providing fault-tolerance for inter-bubble communication without requiring sequential processing. Intermediate results are stored in durable storage between bubbles, enabling parallel execution while maintaining reliability. This mediator approach allows performance improvement through parallelism while preserving fault-tolerance through persistent storage.
2Productivity
If gang scheduling is used to schedule all vertices at once, then performance is improved, but fault-tolerance deteriorates
Solution Approach 1:
The job graph is divided into multiple bubbles that can be scheduled independently rather than requiring all vertices to be scheduled simultaneously. Each bubble can be executed when resources are available, providing both parallelism for performance and isolation for fault-tolerance. This segmentation eliminates the all-or-nothing constraint of gang scheduling.
Solution Approach 2:
The scheduling approach transitions from static gang scheduling (all vertices at once) to dynamic bubble-based scheduling where bubbles are executed when resources are available. This dynamic approach allows the system to adapt to resource availability while maintaining fault-tolerance through bubble boundaries and durable storage for inter-bubble results.
3Productivity
If gang scheduling requires all resources to be available, then performance is improved, but adaptability deteriorates
Solution Approach 1:
The job graph is segmented into multiple bubbles that can be scheduled independently based on resource availability. This segmentation allows the system to execute bubbles when resources are available rather than requiring all resources upfront, improving adaptability to varying resource conditions while maintaining performance through parallel execution of multiple bubbles.
Solution Approach 2:
The scheduling system periodically checks for resource availability and executes bubbles when resources are available, rather than requiring all resources to be available at a single point in time. This periodic scheduling approach provides both performance improvement through parallelism and adaptability to dynamic resource conditions.
Data Source
AI summary
Aspects extend to methods, systems, and computer program products for optimally pipelining result sets with fault tolerance in distributed query execution. Distributed computing jobs are optimized by dividing the distributed computing jobs into one or more bubbles for execution. Each bubble can be independently executed, potentially in parallel with other bubbles, when resources to handle the bubble are available. Intra-bubble communication can be streamed between vertices within a bubble. Inter-bubble communication can be stored to durable storage. Bubbles provide a failure boundary for a job graph and re-executing a bubble along with storage of intermediate results in durable storage can be used to recover from failures. When a vertex inside a bubble fails, computation can resume by rescheduling the execution of the failed bubble from the durable inputs for that bubble. Durable storage provides a light-weight failover to handle non-deterministic behavior. Jobs can also leverage streaming to increase performance.