Distributed Database Failover via Task State Logging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed database systems, failures at scheduler or worker nodes often lead to the abortion of queries, especially during later phases, resulting in costly re-execution of the entire query.
Innovation Solution
A distributed database system design that includes local scheduler nodes and worker nodes maintaining logs of task completion states, allowing for failover handling by coordinating the completion of tasks through notification and log-based reconstruction of execution states, enabling continuation of query execution without re-executing completed tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system aborts queries upon node failure, then system reliability is improved by preventing inconsistent states, but productivity deteriorates due to costly full query re-execution
Solution Approach 1:
The system performs preliminary actions by maintaining detailed logs of task completion states and intermediate data before failures occur. When a node fails, these pre-maintained records enable the system to reconstruct execution state and resume queries without full re-execution, thus resolving the contradiction between reliability and productivity
Solution Approach 2:
The system creates copies of execution state information through logging mechanisms that record task completion status and intermediate results. These copies are stored in a recoverable format that allows reconstruction of the execution state after failures, enabling query resumption while maintaining system reliability
2Reliability
If the system re-executes completed tasks after node failure, then reliability is improved by ensuring consistent results, but loss of time increases due to redundant execution
Solution Approach 1:
The system implements feedback mechanisms through logging that continuously track and record task completion states and intermediate data. This feedback information is used after failures to determine which tasks have already been completed, allowing the system to avoid re-executing them and thus reducing time loss while maintaining result consistency
Solution Approach 2:
By preliminarily recording task completion status and intermediate results in logs, the system prepares the information needed to avoid redundant re-execution. This preliminary action enables quick determination of which tasks can be skipped, reducing time loss while ensuring reliability
3Productivity
If the system reassigns in-progress tasks to other worker nodes, then productivity is improved by continuing query execution, but reliability deteriorates due to potential loss of intermediate data
Solution Approach 1:
The system creates copies of intermediate data and task state information through logging before reassignment occurs. These copies are stored in a durable format that can be retrieved after failures, allowing the system to safely reassign tasks to other worker nodes while preserving data integrity and enabling recovery if needed
4Productivity
If the system maintains detailed logs of task completion states, then productivity is improved by enabling query resumption, but device complexity increases due to additional logging infrastructure
Solution Approach 1:
The logging infrastructure is designed to serve multiple functions: tracking task completion states, recording intermediate data, maintaining execution state for recovery, and providing audit information. This multi-functionality reduces the need for separate specialized systems, thereby limiting the increase in device complexity while enabling query resumption
Data Source
AI summary
In one respect, there is provided a distributed database system that includes a first local scheduler node and a second local scheduler node. The first local scheduler node can be configured to coordinate a performance of a first task and a second task by a first worker node associated with the first local scheduler node. The second local scheduler node can be configured to respond to a defect at the first local scheduler node by at least: collecting, from the first worker node, a log indicating one or more tasks completed by the first worker node prior to the defect at the first local scheduler node; determining, based at least on the log, that the first worker node has completed the first task but not the second task; and coordinating a performance of the second task by the first worker node. Related methods and articles of manufacture are also provided.