Daisy Chain Data Distribution for Large File Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data distribution methods in data centers, such as BitTorrent and HTTP, are inefficient for transferring large files and lack effective failure recovery mechanisms, especially in HDFS systems.
Innovation Solution
Implementing a daisy chain technique using multiple TCP connections to sequentially stream data blocks between nodes, with a failure recovery mechanism that re-transmits missing blocks and allows failed nodes to rejoin the chain upon recovery, reducing dependence on seeders and improving scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If BitTorrent or HTTP protocols are used for data distribution, then data can be transferred across the network, but the transfer speed is insufficient for large files and the system lacks effective failure recovery mechanisms
Solution Approach 1:
The patent segments large data files into smaller blocks and distributes them through a daisy-chain topology where each node receives and forwards blocks sequentially. This segmentation enables faster parallel transfer of multiple blocks while maintaining reliability through block-level error handling and retransmission at the block level rather than full file retransmission.
Solution Approach 2:
The patent introduces intermediary nodes in the daisy chain that act as both receivers and forwarders. Each intermediary node receives blocks from its predecessor, stores them temporarily, and forwards to its successor. This intermediary structure enables pipelining and provides multiple paths for failure recovery, as any node can retransmit blocks to subsequent nodes if failures occur.
2Adaptability or versatility
If traditional data distribution methods are used, then existing systems can maintain compatibility, but scalability is limited and dependency on seeders creates bottlenecks
Solution Approach 1:
The patent implements self-service by enabling each node in the daisy chain to autonomously receive, store, and forward data blocks without requiring continuous coordination with the original seeder. Once a node receives blocks, it independently forwards them to the next node, eliminating the single-point dependency on the seeder and enabling the system to scale to many more nodes than traditional seeder-based systems.
Solution Approach 2:
The patent transitions from the traditional star topology (all nodes connected to a central seeder) to a linear daisy-chain topology. This dimensional change in the network structure allows data to flow through multiple paths and enables any node to potentially serve as a source for recovery, fundamentally improving scalability while reducing seeder dependency.
3Reliability
If large files are transferred using existing protocols, then data distribution can occur, but the process is slow and lacks robust failure recovery mechanisms
Solution Approach 1:
The patent applies preliminary action by pre-establishing the daisy-chain topology and block segmentation before data transfer begins. Each node is pre-configured with its position in the chain and the blocks it will receive and forward. This preliminary setup enables immediate high-speed transfer upon initiation and allows for quick failure recovery since the topology and block assignments are already determined, avoiding the need for complex dynamic reconfiguration during transfer.
Data Source
AI summary
A method and a system to provide daisy chain distribution in data centers are provided. A node identification module identifies three or more data nodes of a plurality of data nodes. The identification of three or more data nodes indicates that the respective data nodes are to receive a copy of a data file. A connection creation module to, using one or more processors, create communication connections between the three or more data nodes. The communication connections form a daisy chain beginning at a seeder data node of the three or more data nodes and ending at a terminal data node of the three or more data nodes.