Multicast Rendezvous for Scalable Storage Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges in achieving reliable multicast delivery of bulk payloads due to limitations in existing protocols, such as high overhead in point-to-point connections and inefficiencies in network traffic management, leading to congestion and data loss.
Innovation Solution
The implementation of a multicast communication protocol using unreliable datagrams over a protected traffic class, which enables edge-managed flow control and congestion avoidance, allowing for efficient and reliable delivery of data chunks across multiple servers without centralized control, thereby optimizing network utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If point-to-point protocols are used for reliable multicast delivery, then delivery reliability is improved, but protocol overhead and network complexity increase exponentially with the number of servers
Solution Approach 1:
The patent merges multiple point-to-point communication channels into a single multicast channel. Instead of establishing separate reliable connections between each server pair, the system uses a single multicast group where all servers receive data simultaneously through one channel, reducing protocol overhead from exponential to linear scaling while maintaining delivery reliability through the multicast paradigm
2Reliability
If centralized control is implemented for traffic management, then flow control and congestion avoidance are improved, but system complexity and maintenance overhead increase
Solution Approach 1:
The patent implements self-service flow control where each server independently monitors its own buffer status and autonomously adjusts its transmission rate. Servers participate in a decentralized negotiation process where they collectively determine the Rendezvous Group composition and traffic parameters without requiring a central controller, thereby achieving effective flow control while reducing system complexity and eliminating single points of failure
3Quantity of substance
If the number of servers in the cluster increases, then storage capacity and processing power are improved, but network congestion and data loss increase due to exponential scaling of connections
Solution Approach 1:
The patent transitions from a two-dimensional point-to-point connection model to a many-to-many multicast dimension. By introducing the multicast group as a new dimensional layer, the system allows a single data transmission to reach multiple servers simultaneously, changing the scaling relationship from exponential (O(n²) connections) to linear (O(n) multicast members), thereby maintaining network reliability as cluster size increases
Data Source
AI summary
Embodiments disclosed herein provide advantageous methods and systems that use multicast communications via unreliable datagrams sent on a protected traffic class. These methods and systems provide effectively reliable multicast delivery while avoiding the overhead associated with point-to-point protocols. Rather than an exponential scaling of point-to-point connections (with expensive setup and teardown of the connections), the traffic from one server is bounded by linear scaling of multicast groups. In addition, the multicast rendezvous disclosed herein creates an edge-managed flow control that accounts for the dynamic state of the storage servers in the cluster, without needing centralized control, management or maintenance of state. This traffic shaping avoids the loss of data due to congestion during sustained oversubscription. Other embodiments, aspects and features are also disclosed.