Task Coordination State Machine for Distributed System Failure Recovery

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed systems face challenges in discovering and recovering from failures due to the lack of task coordination, leading to costly and error-prone manual recovery processes that rely heavily on engineering resources and are prone to violating data integrity regulations.

Innovation Solution

Implementing a task coordination state machine within the distributed system to bind units of work together, enabling the processing chain to be stateful and facilitating error detection, reporting, and automated recovery through a failure recovery service and resolvers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual recovery processes are used in distributed systems, then engineering resources can handle failures, but the process becomes costly, error-prone, and violates data integrity regulations

Engineering Contradiction:

Improvefailure recovery reliabilityVSAvoidrecovery process complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent implements preliminary action by capturing stack traces and system state information at the point of failure before the system needs to recover. This pre-captured information is stored in a structured format that enables automated recovery without requiring manual engineering intervention later, thus improving reliability while reducing complexity

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an intermediary component (failure recovery service) that acts as a mediator between the failure occurrence and the recovery process. This service automatically processes failure information, determines appropriate recovery actions, and executes them without requiring human engineers to directly manage the complex recovery process, thereby reducing operational complexity while maintaining reliability

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If retrying the processing chain is used for failure recovery, then the system can attempt to recover from failures, but it requires significant processing power and is expensive

Engineering Contradiction:

Improvefailure recovery capabilityVSAvoidprocessing power consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary action by capturing and storing the state of the processing chain at each stage before failure occurs. When a failure is detected, the system uses this pre-captured state information to resume processing from the exact point of failure rather than retrying the entire processing chain, significantly reducing processing power consumption while maintaining recovery capability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent extracts the essential state information from the processing chain at each stage and stores it separately. This extracted state information can be used to quickly restore the system to a known good state without re-executing unnecessary processing steps, thereby reducing the processing power required for recovery while maintaining reliability

Inventive Principle:

Principle #2Taking out (Extraction)

3Loss of information

If log files are used to track system state, then information is recorded, but accessing this information requires additional executable code and auditing

Engineering Contradiction:

Improvesystem state trackingVSAvoidcode complexity for accessing logs

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent merges the state tracking functionality directly into the normal processing chain execution. The same code that performs business logic also captures and records state information in a structured format, eliminating the need for separate logging code and auditing mechanisms. This reduces code complexity while maintaining complete system state tracking

Inventive Principle:

Principle #5Merging (Combining)

4Reliability

If executable code is written to access log information for recovery, then the system can utilize logged data, but it is extremely error-prone and requires heavy reliance on engineer understanding

Engineering Contradiction:

Improverecovery accuracyVSAvoidease of recovery operation

Core Design Contradiction:

ReliabilityVSEase of operation

Solution Approach 1:

The patent implements self-service by designing a system where the failure recovery process automatically uses the pre-captured state information without requiring engineers to write or maintain complex access code. The system serves itself by having the failure recovery service directly utilize the structured state data that was automatically captured during normal operation, eliminating human error while maintaining recovery accuracy and ease of operation

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS10360095B2Methods and apparatuses for improving failure recovery in a distributed system

Publication Date: 2019.07.23 CHANGE HEALTHCARE HOLDINGS LLC

AI summary

Embodiments are disclosed for improving failure recovery in a distributed system during execution of a processing chain. In the context of a method, an example embodiment includes implementing, by the distributed system, a task coordination state machine corresponding to the processing chain. This example embodiment of the method further includes discovering, using the task coordination state machine, an error occurring during execution of the processing chain. Finally, the example embodiment of the method further includes reporting the error by a failure report generator service and recovering, by a failure recovery service, from the error. Corresponding apparatuses and computer program products are also provided.