Multicast Transport Protocol for Scalable Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in scalable and reliable data replication due to the limitations of point-to-point protocols, which lead to congestion and increased overhead, especially as the number of storage servers grows, and multicast protocols have not been effectively utilized for reliable bulk payload delivery in storage clusters due to challenges with error handling and congestion control.
Innovation Solution
The implementation of a multicast transport protocol using unreliable datagrams over a protected traffic class, which enables edge-managed flow control and congestion avoidance, allowing for reliable data delivery while reducing network overhead and scaling linearly with the number of servers, and a distributed object storage system that decentralizes processing and assigns responsibilities dynamically based on server capacity and workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If point-to-point protocols are used for data replication, then reliable data delivery is achieved, but network congestion and overhead increase as the number of storage servers grows
Solution Approach 1:
The patent merges multiple unicast data streams into a single multicast stream, allowing one source server to simultaneously replicate data to multiple storage servers through a single network transmission. This combining of communication channels reduces network congestion and overhead while maintaining reliable delivery through protocol-level acknowledgments and retransmissions.
Solution Approach 2:
The multicast transport protocol serves multiple functions simultaneously: it enables efficient one-to-many data distribution, provides reliable delivery guarantees through acknowledgments, handles congestion control, and supports dynamic group membership. This multi-functionality replaces the need for multiple separate point-to-point connections.
2Productivity
If multicast protocols are used for data replication, then network overhead is reduced, but error handling and congestion control become challenging
Solution Approach 1:
The patent introduces intermediary components including a multicast transport layer with acknowledgment mechanisms and congestion control algorithms that mediate between the simple multicast transmission and the reliable delivery requirements. These intermediaries handle error detection, retransmission coordination, and flow control without requiring complex changes to the underlying multicast protocol.
Solution Approach 2:
The system implements feedback mechanisms where receiving servers send acknowledgments back to the source server, and the source server adjusts transmission rates based on network conditions and reception status. This feedback loop enables reliable delivery and congestion control while maintaining the efficiency benefits of multicast transmission.
3Quantity of substance
If the number of storage servers is increased, then storage capacity is improved, but system overhead and complexity increase with point-to-point protocols
Solution Approach 1:
The patent implements dynamic multicast group management where servers can dynamically join and leave the storage cluster without requiring reconfiguration of data replication paths. The multicast protocol automatically adapts to changing group membership, and the system dynamically adjusts replication strategies based on current server capacity and workload conditions.
Solution Approach 2:
The system segments the storage cluster into logical multicast groups based on data types, replication factors, and server capabilities. This segmentation allows different multicast streams to operate independently with optimized parameters for each group, reducing overall system complexity while enabling scalable capacity expansion.
Data Source
AI summary
Embodiments disclosed herein provide a scalable multicast transport. The multicast transport protocol provides effectively reliable multicast delivery while avoiding the overhead associated with point-to-point protocols. Additional embodiments disclosed herein relate to a scalable object storage system that uses a multicast transport. The object storage system assigns responsibility for providing storage services for a chunk to a negotiating group of storage servers in the cluster using a shared and distributed hash allocation table. The object storage system dynamically determines a rendezvous group of storage servers in the cluster to store the chunk using the multicast transport. Other embodiments, aspects and features are also disclosed.