Data Relay Node for Parallel Computing I/O Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In high-performance computing systems, the increasing data volume and communication frequency between computing nodes and file servers lead to a bottleneck in data transfer, causing delays and inefficiencies due to individual data access and communication by each computing node.
Innovation Solution
A data relay node is selected based on the centroid position calculation, which collectively handles data transfer requests from multiple computing nodes, reducing the number of communications and load on file servers through Collective I/O processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If each computing node individually reads or writes data from or to file servers, then data access flexibility is improved, but the number of communications increases and transfer processing is delayed
Solution Approach 1:
A data relay node is introduced as an intermediary between computing nodes and file servers. The relay node receives data from multiple computing nodes, performs collective I/O processing, and communicates with file servers, thereby reducing the number of direct communications while maintaining data access flexibility for all nodes.
Solution Approach 2:
Multiple individual data access requests from different computing nodes are merged into a single collective I/O operation. The system combines data from multiple sources at the relay node and performs unified communication with file servers, improving transfer efficiency while preserving individual node access capabilities.
2Productivity
If communication with file servers is performed by only a specific computing node, then the number of communications is reduced, but the data transfer load increases for nodes far from the communicating node
Solution Approach 1:
The system selects a data relay node based on its spatial position and proximity to other computing nodes. Nodes closer to the relay node experience shorter data transfer distances, optimizing local performance. The relay node is strategically chosen to minimize overall communication distances across the network.
Solution Approach 2:
The centroid position calculation ensures that the selected relay node is optimally positioned relative to all computing nodes, creating a balanced communication topology. This positioning minimizes the variation in transfer distances, making the system more equitable in terms of data access time for all nodes.
3Power
If the number of computing nodes increases, then processing power is improved, but the load on file servers and communication overhead increase
Solution Approach 1:
The system segments the communication burden by introducing a dedicated data relay node that handles collective I/O operations. This segmentation separates individual node access requirements from file server communication, allowing the system to scale to more computing nodes without proportionally increasing file server communication overhead.
Solution Approach 2:
The data relay node serves multiple functions: receiving data from multiple computing nodes, performing collective I/O processing, and communicating with file servers. This multi-functional approach allows a single node to handle communications for the entire computing cluster, reducing overall system complexity despite the increase in computing nodes.
Data Source
AI summary
An information processing apparatus, among a plurality of information processing apparatuses that performs parallel computing processing in a parallel computer system, including a memory and a processor coupled to the memory and configured to execute a process including: calculating a centroid position of the information processing apparatuses based on a data length of data for which subsequent reading or writing from or to a file server is requested by the information processing apparatuses and position information on each of the information processing apparatuses; determining a first information processing apparatus that performs data relay according to the calculated centroid position; and collectively receiving or transmitting, when the determined first information processing apparatus that performs data relay is the information processing apparatus, the data for two or more of the information processing apparatuses.