Write Pacing System for Storage Latency and Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication systems, such as synchronous RDF, suffer from increased latency due to acknowledgement delays, especially over long distances, and lack guaranteed data consistency in case of failures, while asynchronous methods may result in data inconsistencies and significant overhead in maintaining mirror copies.
Innovation Solution
Implement a write pacing system that controls the response time of write requests between storage devices based on system parameters like lag time and queue length, using credits to manage data transmission and ensuring data consistency by associating writes with chunks and committing them in a sequential manner.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous RDF data transfer is used, then data consistency between primary and secondary storage devices is guaranteed, but write latency increases due to waiting for acknowledgement from the secondary storage device
Solution Approach 1:
The patent segments the data transfer process into discrete chunks with sequence numbers, allowing the primary storage device to send multiple chunks in parallel without waiting for individual acknowledgements. This segmentation enables batch processing where groups of chunks can be transmitted together, reducing the cumulative latency while maintaining consistency through sequential numbering and acknowledgment tracking.
Solution Approach 2:
The patent implements preliminary actions by pre-establishing chunk sequences and preparing data blocks before actual transmission is needed. The primary storage device can pre-process and queue multiple data chunks, so when transmission begins, the data is already organized and ready to be sent in efficient batches, reducing the overall time required for data replication.
2Loss of time
If asynchronous data transfer is used, then write latency is reduced, but data consistency between primary and secondary storage devices cannot be guaranteed
Solution Approach 1:
The patent implements a feedback mechanism where the secondary storage device sends acknowledgements back to the primary storage device for received and committed chunks. This feedback loop allows the primary device to track which chunks have been successfully replicated and maintain consistency by knowing the current state of data synchronization, while still allowing asynchronous operation to reduce latency.
Solution Approach 2:
The system dynamically adjusts its operation mode based on conditions. While fundamentally asynchronous to reduce latency, it can switch to synchronous behavior when consistency is critical by adjusting acknowledgment requirements or transmission pacing, making the system flexible enough to handle different operational requirements without being locked into a single mode.
3Loss of time
If background copy mechanism is used, then write latency is attenuated, but data consistency and order of changes cannot be guaranteed in case of failures
Solution Approach 1:
The patent applies preliminary action by pre-numbering data chunks with sequence numbers before transmission. This sequencing is established in advance, creating a predetermined order for data replication. In case of failures, the secondary storage device can use these sequence numbers to reconstruct the correct order of changes, ensuring consistency without requiring continuous background monitoring or complex coordination mechanisms.
4Reliability
If synchronous RDF transfer is used over long distances, then data consistency is maintained, but transmission delays make the system unacceptable
Solution Approach 1:
The patent segments large data transfers into smaller chunks that can be transmitted in parallel batches. This segmentation reduces the impact of long-distance transmission delays by allowing multiple chunks to be in transit simultaneously, effectively hiding latency through parallelization. The secondary storage device processes chunks as they arrive rather than waiting for complete data sets, maintaining consistency while improving effective transfer speed over long distances.
Solution Approach 2:
The system uses periodic batch transmission where groups of chunks are sent at regular intervals rather than one-by-one. This periodic action creates a rhythm of transmission that optimizes bandwidth utilization and reduces the cumulative effect of transmission delays, making long-distance replication more acceptable by establishing predictable performance characteristics.
Data Source
AI summary
A system for controlling the pacing of host data writes in response to changing system conditions allows for the use of variably controlled delays that facilitate asynchronous replication with bounded lag and smooth steady-state performance. Adding delay to the writes slows down the host and dampens activity spikes. A first storage device receives a write request, transmits data to a second storage device, and acknowledges the write request. An amount of additional write response time is controlled between when the write request is received and when the write request is acknowledged by the first storage device, where the amount of additional write response time is controlled according to system parameters including an amount of lag time between when the write request is received and when the write request is committed at the second storage device.