Worker Device Local Dataset Storage for Cluster Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed cluster computing, large datasets require frequent network transmission and memory duplication across multiple applications, leading to increased latency and memory usage, especially when updates are infrequent or small, which can negatively impact performance.
Innovation Solution
A worker device in a cluster stores datasets locally and receives updates via a messaging system, sharing the dataset among applications and ceasing execution if no updates are received within a threshold time, thereby reducing network traffic and memory duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the master device sends the dataset to each worker device for every task, then the worker devices can access the data quickly, but the network traffic and memory usage increase significantly
Solution Approach 1:
The worker device proactively subscribes to datasets and stores them in local storage media before they are needed for tasks. This preliminary action eliminates the need for repeated network transmissions, as the data is already available locally when required.
Solution Approach 2:
The system creates a local copy of the dataset on the worker device's storage media. This copy allows multiple applications to access the data without requiring repeated network transmissions from the master device, significantly reducing network traffic.
2Speed
If the dataset is copied to memory for each application, then each application can access data quickly, but the overall memory usage is multiplied by the number of applications
Solution Approach 1:
Multiple applications share a single copy of the dataset stored in local storage media on the worker device. Instead of each application having its own private copy in memory, they all access the same shared resource, eliminating redundant memory usage while maintaining fast access speeds.
3Reliability
If the worker device frequently checks for dataset updates, then the data remains current, but the performance decreases due to latency from constant communication
Solution Approach 1:
The worker device subscribes to dataset updates and receives notifications only when changes occur. This feedback mechanism allows the system to maintain current data without requiring frequent polling checks, eliminating unnecessary communication overhead and latency while ensuring data currency.
Data Source
AI summary
Methods and systems for delivering data for cluster computing are described herein. A worker device may receive a dataset and store the dataset in a local storage media. This may prevent the need for the dataset to be sent over a network each time the applications are used to perform a task. Each application may be able to access the dataset in the local storage area. This may prevent the need to copy the dataset to memory associated with each application. A worker device may store a dataset, for example, if it determines that the frequency of updates to the dataset satisfy a threshold. The worker device may receive updates to the dataset via a messaging system and may store the updated data in the local storage media.