Distributed Computing Task Assignment via Storage Node Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing for geophysics algorithms, the high data access requirements often lead to link overloading and significant computation time, particularly in intensive computations like seismic processing, where data reading can account for up to 80% of the total time.
Innovation Solution
The method involves dividing computing clients into groups with a storage node each, determining task associations to minimize the size of the union of data subsets, and optimizing data retrieval by using I/O nodes to reduce data access requests to the storage server, allowing for efficient data reuse and caching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is retrieved from storage server for each computing client to perform computing tasks, then computing tasks can be executed, but link overloading and significant computation time occur due to high data access requirements
Solution Approach 1:
The patent divides computing clients into groups with dedicated storage nodes, segmenting the data access architecture. Each group retrieves data once from the storage node rather than each client retrieving data individually, reducing redundant data access operations and decreasing overall computation time.
Solution Approach 2:
The patent introduces storage nodes as intermediary components between computing clients and the central storage server. These storage nodes cache data locally for groups of clients, acting as mediators that reduce the burden on both the central storage server and individual client connections, thereby preventing link overloading.
2Productivity
If multiple computing clients access data from storage server simultaneously, then computing tasks can be performed, but link overloading occurs
Solution Approach 1:
The patent segments clients into groups with dedicated storage nodes, distributing data access load across multiple storage nodes rather than concentrating all client requests on a single storage server. This segmentation prevents link overloading by spreading traffic across multiple pathways.
Solution Approach 2:
The patent implements data caching by copying data from the central storage server to local storage nodes. This copying allows multiple clients to access data simultaneously from the local cache without overloading the original storage server links, as the cached data can be served locally.
3Speed
If data reading speed is increased to reduce computation time, then computing efficiency improves, but storage server read speed becomes a limiting operation
Solution Approach 1:
The patent performs preliminary data retrieval by having storage nodes fetch and cache data before computing tasks are executed. This preliminary action prepares data in advance, allowing fast local access during computation without subjecting the storage server to high read speed demands during task execution.
Solution Approach 2:
The patent copies data to local storage nodes, creating local copies that can be read at high speeds without bottlenecking the central storage server. This copying decouples the read speed requirement from the storage server, allowing fast data access while the storage server operates at its own optimized read speed.
Data Source
Figure 1
Figure 2~4b
Figure 3
AI summary
The present invention relates to a method for computing tasks by computing clients. The method comprises distributing tasks to the clients; the determination of the association between distributed task and clients is based on a minimization of a size of a union of the subsets of data associated with tasks distributed to computing clients of at least one group in the plurality of clients groups. In addition the method further comprise, for each current computing client in the plurality of computing clients retrieving the subset of data associated with the task distributed to said current computing client from the storage node of the group of the current computing client, and computing the task distributed to said current computing client based on said retrieved subset of data associated with the task distributed to said current computing client.