Cluster Data Distribution via Segmented Concurrent Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale cluster environments, the incremental deployment technique for distributing large-sized data, such as configuration files and images, from multiple servers to multiple nodes results in bottlenecks due to limited node bandwidth and sequential data transfer, leading to inefficient use of server bandwidth and prolonged deployment times.
Innovation Solution
Divide data on each server into N segments and send these segments concurrently to N nodes, with each node transmitting received data segments to other nodes in the cluster, thereby utilizing server bandwidth more efficiently and reducing deployment time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is distributed sequentially from multiple servers to multiple nodes, then node bandwidth limitations are respected, but server bandwidth is not fully utilized and deployment time is prolonged
Solution Approach 1:
The patent divides data into multiple segments and distributes them concurrently from multiple servers to multiple nodes simultaneously. This segmentation allows parallel data transfer operations, fully utilizing server bandwidth while maintaining manageable data flow through each node, thereby resolving the contradiction between deployment speed and distribution time.
Solution Approach 2:
The patent implements continuous concurrent data transfer operations where multiple servers continuously send data segments to multiple nodes simultaneously without sequential waiting. This continuous parallel action maximizes server bandwidth utilization and significantly reduces overall deployment time compared to sequential transfers.
2Productivity
If multiple servers send data to multiple nodes concurrently, then server bandwidth utilization improves, but node bandwidth becomes a bottleneck
Solution Approach 1:
By dividing data into segments and assigning specific segments to specific server-node pairs, the patent distributes the bandwidth burden across multiple nodes rather than overwhelming single nodes. This segmentation strategy enables concurrent transfers while managing node bandwidth loads effectively.
Solution Approach 2:
The patent introduces a coordination mechanism that acts as an intermediary to manage and coordinate the concurrent data transfers between multiple servers and nodes. This mediator optimizes the distribution strategy to balance server bandwidth utilization with node bandwidth capacity, reducing the complexity of direct peer-to-peer coordination.
3Adaptability or versatility
If incremental deployment is used to deploy from multiple servers, then deployment scalability is improved, but data transfer efficiency decreases due to sequential operations
Solution Approach 1:
The patent combines incremental deployment scalability with segmentation by dividing data into segments that can be independently distributed. This allows the system to scale to multiple servers and nodes while performing concurrent segment transfers, maintaining both scalability and high data transfer efficiency without sequential bottlenecks.
Solution Approach 2:
The patent transitions from sequential one-dimensional deployment to concurrent multi-dimensional deployment by distributing data segments across multiple server-node pairs simultaneously. This dimensional expansion allows incremental scalability while achieving parallel transfer efficiency, resolving the contradiction between adaptability and productivity.
Data Source
AI summary
The embodiments of the present disclosure disclose a computer-implemented method, a system, and a computer program product for distributing data on multiple servers to multiple nodes in a cluster. In the method, each of M servers is instructed to divide data thereon into N data segments. M and N are integers greater than one. The M servers are instructed to send N×M data segments on the M servers to N nodes in a cluster concurrently. For each of the M servers, the N data segments are sent respectively to the N nodes. When any given node in the cluster receives a data piece of a data segment from a server of the M servers, the given node is instructed to transmit the received data piece to remaining nodes in the cluster other than the given node.