Distributed Processing Task State Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed processing systems like MapReduce require re-processing of completed Map tasks upon node failure, leading to delays due to the lack of stored execution states, and do not provide flexibility in achieving high reliability.
Innovation Solution
A distributed processing system that includes an object storage unit for storing continuation objects containing task processes and data, a processing unit to execute these objects, and a storage processing unit to store execution state files, allowing for the retrieval and storage of task states, enabling flexible and reliable processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If MapReduce stores input-output results in a distributed file system, then reliability is improved, but the system cannot return to execution state at a point in time when task was in progress, causing delay in system recovery
Solution Approach 1:
The patent applies preliminary action by storing execution state files at predetermined intervals during task execution. This allows the system to capture the state of tasks at specific points in time before failure occurs, enabling rapid recovery without requiring complete re-execution of all tasks. The execution state files are prepared in advance and can be immediately utilized when failure is detected.
Solution Approach 2:
The patent implements copying by creating execution state files that replicate the task execution state at different time points. These state files are stored in the distributed file system and can be retrieved when needed for recovery. The copying mechanism allows the system to preserve execution states without altering the original task execution flow, enabling selective restoration to any captured state point.
2Reliability
If MapReduce re-processes all completed Map tasks upon node failure, then reliability is maintained, but processing time increases significantly
Solution Approach 1:
The patent extracts the essential recovery information from the complete task execution history by storing only the critical execution state files at predetermined intervals. When failure occurs, the system retrieves only the necessary state information from these extracted files rather than re-processing all tasks, significantly reducing recovery time while maintaining reliability.
Solution Approach 2:
The patent applies partial action by storing execution states at specific time points rather than continuously or completely. This selective storage approach captures sufficient information for recovery without the overhead of storing every possible execution state, balancing reliability requirements with processing efficiency.
3Device complexity
If MapReduce does not store execution states of tasks, then device complexity is reduced, but the system lacks flexibility in recovering to specific execution states
Solution Approach 1:
The patent implements preliminary action by establishing a mechanism that automatically captures and stores execution states at predetermined intervals during normal operation. This proactive approach prepares recovery resources in advance without adding significant complexity to the execution flow, as the state storage occurs as a byproduct of routine task processing.
Solution Approach 2:
The patent introduces execution state files as intermediary structures that bridge the gap between task execution and recovery operations. These state files serve as mediators that preserve execution information without requiring complex state management mechanisms, enabling flexible recovery to any captured state point while maintaining relatively simple system architecture.
Data Source
AI summary
A distributed processing device includes an object storage unit that stores a continuation object including at least one of plural processes constituting a task and containing data of the task that is being processed, a processing unit that executes the continuation object retrieved from the object storage unit, and a storage processing unit that stores, in an execution state file, data stored in the object storage unit.