Distributed Shared Memory Failover for Query Nodes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In distributed query processing, failures in query execution lead to costly re-execution of the entire query, resulting in doubled resource requirements and time, which is inefficient and expensive, especially in real-time computing systems with mission-critical performance needs.

Innovation Solution

Implementing automatic recovery using distributed shared memory, where worker nodes save their state at intervals, and idle nodes monitor and resume execution upon failure detection, employing a lock-based agreement scheme to ensure seamless failover without re-executing the entire query.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If query execution fails on any evaluator node, then the query must be re-executed completely, but this results in doubled resource requirements and execution time

Engineering Contradiction:

Improvequery execution reliabilityVSAvoidquery execution efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system performs preliminary actions by saving the state of worker nodes at predetermined time intervals during query execution. This checkpointing mechanism ensures that if a failure occurs, the system can resume from the last saved state rather than re-executing the entire query, thus improving reliability without sacrificing productivity

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates copies of the worker node states at predetermined intervals and stores them in distributed shared memory. These state copies enable failure recovery by allowing idle nodes to resume execution using the copied state, avoiding the need to re-execute the entire query and maintaining high productivity

Inventive Principle:

Principle #26Copying

2Reliability

If the entire query is re-executed after failure, then query execution can be completed, but the cost and time required double

Engineering Contradiction:

Improvefailure recovery capabilityVSAvoidquery execution time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary state saving at predetermined time intervals during query execution. When failure occurs, the system uses these pre-saved states to resume execution, avoiding the time penalty of complete re-execution and reducing the loss of time while maintaining reliability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts query execution by allowing idle nodes to take over failed worker nodes and resume execution from saved states. This dynamic failover mechanism enables the system to adapt to failures without static re-execution, significantly reducing time loss while ensuring reliable query completion

Inventive Principle:

Principle #15Dynamics

3Reliability

If worker node state is saved in distributed shared memory, then failure recovery is enabled, but system complexity increases

Engineering Contradiction:

Improveautomatic failure recoveryVSAvoidsystem architecture complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The distributed shared memory serves multiple functions: it stores worker node states for failure recovery, enables communication between nodes, and supports the failover mechanism. This multi-functionality reduces the need for separate dedicated components, thereby limiting the increase in system complexity while enabling automatic failure recovery

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The distributed shared memory acts as an intermediary between worker nodes and idle nodes, facilitating state transfer and failover. By introducing this intermediary, the system achieves automatic failure recovery without requiring complex direct communication protocols between all nodes, thus limiting complexity increase

Inventive Principle:

Principle #24Intermediary (Mediator)

4Reliability

If idle nodes monitor worker nodes by polling shared memory, then failure detection is achieved, but message transmission overhead increases

Engineering Contradiction:

Improvefailure detection accuracyVSAvoidmessage transmission overhead

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

Idle nodes monitor worker nodes by polling the shared memory at predetermined time intervals rather than continuously. This periodic monitoring achieves reliable failure detection while significantly reducing message transmission overhead and energy consumption compared to continuous monitoring

Inventive Principle:

Principle #19Periodic action

Data Source

PatentUS8874961B2Method and system for automatic failover of distributed query processing using distributed shared memory

Publication Date: 2014.10.28 INFOSYS LTD

AI summary

A method and system for implementing automatic recovery from failure of resources in a grid-based distributed database is provided. The method includes determining the category of each node in the subgroup of nodes, where the determination identifies each node as at least one of a worker node and an idle node. The method further includes saving state of each worker node engaged in execution of a task in a shared memory at pre-determined time intervals. Each worker node is monitored by one or more idle nodes in each sub-group. Upon detection of no change in state of worker node for a pre-determined period of time, a failure notification is raised by one or more idle nodes that have detected failure of the worker node.