Distributed Processing System with Worker Node Data Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed processing systems, the existing data replication techniques lead to performance deterioration in the steady state due to load concentration on the job scheduler and time lag in synchronization processing during data updates, causing inefficiencies in failure recovery.
Innovation Solution
A distributed processing system with worker nodes that cache output data and copy it to another node, while a master node manages worker node identifiers to optimize task allocation and data placement, reducing the load on the scheduler and enabling rapid failure recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If notification processing to job scheduler occurs every time data replication processing associated with data update occurs, then failure recovery time is shortened, but load concentration on job scheduler and performance deterioration in steady state occur
Solution Approach 1:
The system performs data replication in advance before failure occurs, so that when a failure happens, the replicated data is already available on standby worker nodes, enabling immediate failure recovery without needing to notify the job scheduler during the recovery process
Solution Approach 2:
The patent extracts the notification processing function from the job scheduler and implements it locally at worker nodes. Each worker node independently tracks and manages its own data replication status, removing the burden of centralized notification processing from the job scheduler and eliminating load concentration
2Reliability
If data replication processing occurs frequently to maintain data availability, then failure recovery capability is improved, but time lag in synchronization processing and performance deterioration occur
Solution Approach 1:
The system implements periodic data replication at worker nodes based on local caching policies rather than continuous synchronization. Each worker node periodically updates its cached data and replicates to standby nodes, reducing synchronization overhead while maintaining data availability
Solution Approach 2:
Worker nodes autonomously manage their own data replication and synchronization without external coordination. Each node independently determines when and what to replicate based on its local cache status and task execution patterns, eliminating synchronization time lags caused by centralized coordination
Data Source
AI summary
The present invention relates to a distributed processing system having a master node and a plurality of worker nodes. Each worker node has an assigned identifier. A worker node caches in its own memory first output data, which is the result of the execution of a first task, and copies said first output data to another worker node. The master node selects, on the basis of the identifier information of the first worker node, a worker node to which to assign a second task, wherein the first output data is used as input data.