Clustered Storage Traffic Control via Replication Throughput Buffering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Clustered storage systems face challenges in maintaining data synchronization, particularly during errors or failures, leading to potential data loss and failure in meeting recovery point objectives due to asynchronous replication processes.
Innovation Solution
The implementation of a clustered system architecture that includes synchronous and semi-synchronous replication methods, with components like splitters, sync transfer engines, and file system controllers, which monitor and control client traffic throughput to ensure timely replication and synchronization between primary and secondary clusters, using buffers to manage data flow and maintain replication throughput equal to or greater than client traffic throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If asynchronous replication is used to store backup information, then storage capacity is improved, but data synchronization reliability deteriorates
Solution Approach 1:
The system dynamically switches between synchronous and asynchronous replication modes based on operational conditions. During normal operations, asynchronous replication provides high storage capacity. Upon detecting failure conditions, the system transitions to synchronous replication mode to ensure data synchronization reliability, thus resolving the contradiction between storage capacity and synchronization reliability.
Solution Approach 2:
The replication mode parameter is changed from static asynchronous to dynamic, allowing the system to adjust the synchronization level based on failure detection. This parameter change enables the system to maintain high storage capacity while ensuring data synchronization reliability when needed through controlled transitions between replication modes.
2Reliability
If synchronous replication is implemented to ensure data synchronization, then data integrity is improved, but client traffic throughput deteriorates
Solution Approach 1:
The replication process is segmented into normal operations mode and failure recovery mode. During normal operations, asynchronous replication is used to maintain high client traffic throughput. Upon failure detection, the system segments the data flow to apply synchronous replication only to critical data paths, thus maintaining data integrity while minimizing impact on overall throughput.
Solution Approach 2:
Instead of applying synchronous replication to all data operations (excessive action), the system applies it partially only when failure conditions are detected. This partial application of synchronous replication ensures data integrity for critical operations while avoiding the throughput penalty for all operations, thus resolving the contradiction between data integrity and client traffic throughput.
3Reliability
If replication throughput is increased to meet recovery point objectives, then data loss prevention is improved, but system complexity deteriorates
Solution Approach 1:
The system implements self-service through automated failure detection and mode switching mechanisms. The replication controller automatically monitors replication throughput, detects failures, and switches between synchronous and asynchronous modes without manual intervention. This self-service capability ensures data loss prevention through appropriate replication modes while minimizing the complexity burden on operators.
Solution Approach 2:
The system employs feedback mechanisms where the replication controller continuously monitors replication throughput and system state. Based on this feedback, the controller automatically adjusts replication mode to maintain data loss prevention while managing system complexity through intelligent, automated control rather than complex manual configurations.
4Reliability
If failover capabilities are enhanced to ensure transparent operation, then system reliability is improved, but control complexity deteriorates
Solution Approach 1:
The replication controller acts as an intermediary between the storage system and client devices, managing failover operations transparently. This intermediary automatically handles the complexity of failover control, switching between primary and secondary storage systems as needed, while presenting a simple, consistent interface to clients. This resolves the contradiction by isolating control complexity within the intermediary layer.
Solution Approach 2:
The failover system implements self-service through automated detection and execution of failover operations. The replication controller continuously monitors system health and automatically initiates failover when needed, eliminating the need for complex manual control procedures. This self-service mechanism enhances system reliability while keeping control complexity manageable through automation.
Data Source
AI summary
Various embodiments are generally directed to an apparatus and method to receive client traffic comprising information at a primary cluster of a clustered system over a communications link, perform, a replication operation on the clustered system to replicate the information on a secondary cluster of the clustered system, and determine a client traffic throughput for the client traffic and a replication throughput for the replication operation. In some embodiments, the apparatus and method may include buffering one or more write operations to control the client traffic such that the client traffic throughput is less than or equal to the replication throughput for the replication operation.